Clubhouse tech stack and reason for glitches
Why am I writing this article?
Out of curiosity. I experienced several glitches at Clubhouse last weekend. The problem seems to be the contact info loading. It prompts me to dive deeper into the tech stack Clubhouse is using.
TL;DR:
💰Operation cost shall be much lower than I estimated last time.
⚙️Lack of profile cache causes a scalability problem.
📱The developer has done some optimizations.
Tech stack
By no means is this a complete list of the tech stack Clubhouse uses. Here are just a few things I noticed. Maybe I will do a deeper dive if there are interests.
Pubnub
Pubnub is the most critical service provider for Clubhouse. The audio chat routes via Pubnub (clubhouse.pubnub.com). It is the entry point for the integration with Agora.
Agora
Clubhouse integrates with Agora via the Pubnub APIs. Therefore, they are paying a feed negotiated with Pubnub instead of Agora. My previous calculation in 3 Days impression of Clubhouse is far too high because I am sure the deal Agora cut with Pubnub is much lower.
AWS
The user profile photo storage is in AWS. It sounds like the system will scale quickly. But what I have experienced was some user profile loading failures. You may wonder if there is another bottleneck somewhere in the system that failed to scale when the weekend traffic hit. Read on to the section about the app itself.
No profile cache
I noticed that there are multiple requests to www.clubhouseapi.com
This is a problem.
- www.clubhouseapi.com does not seem to scale well.
- When I resume the app from the background, three connections are established. Because it is an audio-only app, it is very natural to switch the app to the background. Think about how many people keep switching the app between foreground and background. It creates a lot of requests.
- It does NOT cache the profile information! The app fetches a lot of data for each profile. I find that a simple empty description profile is about 3.7 KB of data. A profile with lots of descriptions can reach 10KB.
Proposed solution:
One obvious solution is to scale the user profile service. Because I don’t have insight into the architecture, it is not possible for me to make a suggestion.
Think from a different angle, how can Clubhouse reduce the amount of data that needs from server? Here is my proposal.
Split the profile into two parts. Relatively static information and dynamic information.
Static information: The description, joined time and “nominated by”.
Dynamic information: Followers and following.
For static information:
- Implement a versioning mechanism.
- Build a Memcache layer to cache the version for each profile.
- When the client asks for a profile, the server first checks if the client version matches the server version.
- If it matches, it returns the same version. The client knows there is nothing updated.
- If not, send the profile back with the new version. The client caches the result.
- The client implements a TTL for the profile. This TTL shall be controllable by a server response. The client will not request for profile if the previous response is within the TTL. It gives Clubhouse control of the trade-off between fresh profiles and server load.
For dynamic information:
I am not sure if it is a problem. Therefore, I am not going to reiterate how to design a following and follower system. This problem has been solved many times. Just some small suggestions for the client experience
- Use client cache first
- If loading failed, it is ok to keep the old counts.
iOS optimizations
The develop did some optimizations. One most obvious is keeping the open connection.
Keeping an open connection
I noticed that Clubhouse uses this a lot. For example, there is a long connection open with www.clubhouseapi.com while browsing user profiles.
Keeping an open connection is a good practice to reduce network latency and save data for users. It saved round trips to establish the connection. For example, a TLS handshake is about 1.25kB if the certificates have been exchanged and about 200ms using my Wifi.
Other things:
I recommend readers also read What’s in the Clubhouse iOS app?. The article provides some information about the library the app uses.