Scaling Unlimited Live Chat: Architecture, Consistent Hashing, and Notification-Pull

Live streaming has a concurrency problem that most infrastructure wasn't designed to solve.

When a major broadcast goes live — a national event, a product launch, a celebrity stream — viewer counts can spike from thousands to millions within minutes. Every one of those viewers expects to see the chat moving in real time. They want to send messages, receive reactions, and feel like they're part of something happening right now. Any lag, any dropped messages, any frozen chat panel — and the immersive experience that makes live streaming valuable starts to break down.

This is a routine engineering challenge for anyone building live streaming products at scale. And it's the problem Nexconn's live chatroom infrastructure was specifically designed to handle.

What a Live Chatroom Actually Has to Do

Before getting into architecture, it's worth being precise about what a production live chatroom needs to support — because the requirements are harder than they first appear.

Multiple message types and controls. Beyond plain text, chatrooms carry emoji, gifts, system announcements, and custom interaction types. Each has different delivery requirements. A gift animation needs to arrive on time or not at all. A moderation action needs guaranteed delivery ahead of everything else.

User management at scale. Chatrooms require the ability to create rooms, join and leave them, ban users, mute specific participants, manage allowlists, and query membership — all while tens of thousands of users might be entering or exiting simultaneously. In a high-energy stream, the join/leave concurrency alone can reach thousands of events per second.

Unlimited concurrent users. Major broadcasts routinely accumulate tens of millions of cumulative viewers, with simultaneous viewer counts in the hundreds of thousands. The chatroom infrastructure needs to handle this without a hard ceiling.

High-throughput message distribution. A chatroom with one million users and a modest message rate of 10 messages per second generates 10 million delivery operations per second. At peak engagement — where users might be sending 200+ messages per second — the math gets extreme very quickly. Message distribution isn't a linear problem; it scales geometrically with room size.

These four requirements interact with each other in ways that create real engineering tradeoffs. Solving for message throughput without solving for user management just shifts the bottleneck. Getting the architecture right means addressing all of them together.

The High-Availability Foundation

Nexconn's Open Channel system is built on a three-layer architecture designed to isolate failure domains and enable independent scaling of each component.

Layer 1: Connection Management

The connection layer manages long-lived TCP connections between clients and servers. Its job is stable, persistent, low-latency connectivity for every viewer. This layer is separated from business logic intentionally — connection management has different scaling characteristics and failure modes than message processing, and conflating the two creates fragility.

Layer 2: Storage

Redis serves as the centralized state storage layer — architected following industry-standard horizontal scaling and high-availability patterns — holding chatroom state that needs to survive service restarts: member lists, allowlists, ban lists, room configuration. The architectural decision to externalize this state — rather than keeping it exclusively in process memory — is what makes graceful service restarts and rolling deployments possible without losing room state mid-broadcast.

When a service node restarts or a new node comes online, it loads chatroom data from Redis before accepting traffic. From the end user's perspective, the restart is invisible.

Layer 3: Business Logic

The business layer is split into two distinct services with separate responsibilities:

The Chatroom Service handles administrative operations: users joining and leaving rooms, bans, mutes, allowlist management, and inbound message validation and moderation. It owns the authoritative state of who is in which room and what rules apply.

The Message Service handles distribution: it caches the user set assigned to each node and manages the message queues that deliver content to connected clients. It's responsible for actually getting messages from the chatroom to the viewers.

This split matters for scaling. Administrative operations (user joins/leaves, moderation actions) are relatively infrequent and can be handled by a modest number of chatroom service nodes. Message distribution is the high-volume operation, and the message service nodes can be scaled independently based on throughput demand.

Multi-availability-zone deployment with Zookeeper-based service discovery rounds out the architecture, providing cross-datacenter failover and enabling service instances to find each other dynamically as the cluster scales.

See how this exact architecture supports high-concurrency voice social platforms in our Azal Live Infrastructure Case Study.

Solving the Unlimited Users Problem

The hardest problem in live chatroom infrastructure is delivering it to everyone.

Consider the math: a single message sent in a chatroom with one million concurrent viewers requires one million delivery operations. If each delivery takes even one millisecond of server time, a single message consumes 1,000 server-seconds of processing. At 200 messages per second, that's 200,000 server-seconds per second — clearly impossible on any single machine.

Nexconn's solution is distribution through consistent hashing.

Sharding Users Across Message Service Nodes

When a user joins a chatroom, consistent hashing on the user ID determines which message service node "owns" that user. At a scale of one million concurrent users spread across 200 message service nodes, each node handles approximately 5,000 users on average.

When a message arrives at the chatroom service, it broadcasts to all message service nodes. Each node then delivers only to the users it owns. A node managing 5,000 users delivers to 5,000 clients — a completely tractable operation for a single server.

The result: one million deliveries happen in parallel across 200 nodes, each doing their share. The system's total delivery capacity scales linearly with the number of message service nodes.

Why Consistent Hashing Specifically

Consistent hashing was chosen over simpler sharding approaches for a specific reason: cache efficiency.

In a consistent hashing scheme, the same user always routes to the same node (barring topology changes). This means chatroom state — membership lists, ban status, message history — accumulates and stays on the same node rather than being spread across different servers on different requests.

The practical consequence: most operations that need to check room state (is this user allowed to send? is this message from a banned user?) can be answered directly from in-process memory, without round-tripping to Redis or any other external store. At the message rates live streaming demands, eliminating those round-trips makes a measurable difference in both latency and throughput.

Smooth Scaling Without Disruption

A system that handles current load is necessary but not sufficient. What matters in live streaming is whether the system can scale up during a broadcast — when viewer counts are growing in real time and redeploying infrastructure isn't an option.

Scaling the Chatroom Service

Chatroom service nodes load their state from Redis on startup, so new nodes come online with full visibility into room membership, bans, and configuration. While Message Service nodes are sharded by User ID, Chatroom Service nodes use consistent hashing on the Room ID to assign authoritative room management. The one non-obvious detail: when a node runs its automatic room cleanup logic (destroying empty rooms on a timer), it first checks whether it is the authoritative node for that room. If it isn't — because the room was re-hashed to a different node during scaling — it skips the cleanup.

Scaling the Message Service

Scaling the message service is more complex because users are cached in process memory at each node, and adding or removing nodes changes which node owns which users.

Nexconn handles this with an activity-driven migration approach rather than a disruptive bulk transfer:

During scale-out: When a message is processed, the message service walks its cached user list and checks whether each user still belongs to this node under the new topology. Users who have migrated are synced to their new node. This happens incrementally, at the cadence of message traffic — more active rooms migrate faster, which is also where the extra capacity is most needed.

During user message pulls: When a client requests messages and the node doesn't have them in its cache, the node queries the chatroom service to verify whether the user is still in the room. If confirmed, the user is added to the node's cache. This handles users who migrated to the node but haven't triggered the active migration path yet.

During scale-in: Nodes being removed pull the full member list from Redis and apply consistent hashing to identify which users they should now own, then populate their local cache accordingly.

The result is a scaling process that degrades gracefully rather than catastrophically — room state is never lost, and the migration cost is amortized across message traffic rather than paid all at once.

Handling Billions of Messages: The Distribution Strategy

With user distribution solved, the remaining challenge is throughput: how to handle hundreds of messages per second in large rooms without messages piling up, introducing unacceptable delay, or overwhelming client devices.

Notification-Pull vs. Push

The fundamental design choice in Nexconn's message delivery is a notification-pull model rather than pure push, leveraging modern low-latency transport efficiency (such as QUIC / RFC 9000) for client-edge communication.

In a push model, the server sends each message directly to each client as it arrives. At scale, this creates problems: the server must maintain per-client send queues, clients can be overwhelmed by sudden bursts, and a slow client can back up server resources.

In Nexconn's notification-pull model, the server sends a lightweight notification signal to clients, and clients request the message batch when they're ready. The detailed flow:

1. A user sends a message; the chatroom service processes it and broadcasts to all message service nodes.
2. Each message service node adds connected users to a pending-notification queue (updating the timestamp if already queued, to coalesce multiple pending messages).
3. The delivery thread cycles through the notification queue, sending one notification per user per cycle — regardless of how many messages have arrived since the last notification.
4. Clients receive the notification and pull messages from the server using their local maximum timestamp, fetching only messages newer than what they've already received. To prevent thundering-herd pull storms when millions of clients receive a notification simultaneously, the client SDK applies randomized micro-jitter (staggered delay) and coalesces consecutive pull triggers before firing the request.

The coalescing in step 2 is significant: if 50 messages arrive before a client pulls, they receive one notification, not 50. The client then pulls all 50 in a single request. This dramatically reduces per-client connection overhead and prevents notification storms during peak message rates.

On first join, clients pass a timestamp of 0 and receive the 50 most recent messages. Subsequent pulls pass their local maximum timestamp for differential updates.

Message Rate Control

Even with the notification-pull model, a chatroom receiving 500 messages per second needs rate limiting — otherwise slow-pulling clients accumulate unbounded queues, and the aggregate delivery state of the room becomes unmanageable.

Nexconn applies rate control at two points:

Inbound rate limiting: The chatroom service enforces a per-room inbound message limit (200 messages per second by default, configurable). Messages exceeding this limit are dropped at the chatroom service before being broadcast to message service nodes. This is a hard protection against rooms that could otherwise saturate server-to-server bandwidth.

Outbound rate limiting: The message service uses a ring buffer for the outbound message queue. When the buffer is full, the oldest (lowest-priority) messages are evicted to make room for new ones. Clients that pull frequently enough receive everything; clients that pull slowly receive the most recent messages and miss older ones.

An additional optimization: when a notification is sent to a user, the system marks that user as "pull in progress." If a new message arrives within 2 seconds and the mark is still set, no additional notification is sent. After 2 seconds, a new notification is issued. This prevents notification storms from accumulating for any individual client while ensuring that missed pulls eventually trigger a retry.

Message Priority Tiers

Not all messages are equal. In a live stream, a moderation action removing a user needs to arrive reliably. A gift animation is important but time-sensitive — a delayed gift is worth less than a timely one. A casual text message is the lowest-stakes delivery.

Nexconn maintains three message priority tiers:

Messages are stored in three separate queues. Clients pull in priority order: critical system messages first, then high, then low. Under heavy load, this means a system notification announcing a stream end will always arrive before the backlog of text chat, regardless of how backed up the low-priority queue is.

Developers configure priority levels through the server API or the management console. The defaults are intentionally conservative — most message types start at high priority, and developers opt specific message types into low priority when they want the system to shed them gracefully under load.

Client-Side: Handling the Volume That Gets Through

Architecture and rate control determine what reaches the client. The client still has to render it without degrading the streaming experience.

A few principles Nexconn applies in its client SDK:

MVVM with strict thread separation. All message processing — deduplication, sorting, formatting — happens on a background thread in the ViewModel. The UI thread is only touched when a complete, renderable update is ready. This prevents message processing from competing with the video player for rendering time.

Selective refresh suppression. When the user is scrolling through chat history, new messages don't trigger a UI refresh — only an indicator that new messages have arrived. Interrupting scroll to re-render the list would break the user's reading flow and waste CPU on content they're not looking at.

Differential updates via DiffUtil. Rather than re-rendering the entire message list when new messages arrive, Android's DiffUtil identifies which specific items changed and updates only those. At 400 messages per second in a test on a mid-range handset, the message list scrolled without frame drops.

Post-room cleanup. Chat history in a live stream has zero value to the user after they leave the room. On exit, the client clears the local message database for that chatroom, keeping storage consumption from growing unboundedly across many stream sessions.

Custom Attributes: Beyond Messages

Live streaming products often need to synchronize state that isn't a message — seat assignments in audio spaces, game state in interactive streams, role assignments in collaborative broadcasts.

Nexconn handles this through a key-value custom attribute system with two storage components:

• Full snapshot: the complete current state of all attributes, used by users joining mid-stream to immediately sync to the current state without replaying history
• Incremental change log: an ordered map of attribute changes indexed by timestamp, used by users already in the room to receive only what's changed since their last sync

This design eliminates the need for clients to compare full snapshots to detect changes — a computationally expensive operation that gets worse as attribute sets grow. A client that last synced at timestamp T requests all changes with timestamp > T and applies them locally. The server never needs to compute a diff.

The distribution mechanism for attribute changes mirrors message distribution: the server sends a notification signal, and clients pull changes using their local maximum timestamp. The same priority and rate-control infrastructure applies, ensuring attribute updates don't compete with critical messages for delivery bandwidth.

The Engineering Tradeoffs Worth Noting

A few honest observations about what this architecture optimizes for and where tradeoffs exist:

Eventual consistency under load. In high-throughput scenarios, low-priority messages can be dropped before reaching clients. This is intentional — the alternative (guaranteed delivery of every message) would require client-side buffering and could delay higher-priority content. For live chat, where the value of a message is almost entirely in its immediacy, dropping an old text message is almost always the right call.

Scaling latency. The activity-driven migration during message service scaling means the system reaches its new steady state over time rather than instantly. For most scaling events (traffic growing over minutes or hours), this is unnoticeable. For extremely rapid spikes — a stream that goes viral in 30 seconds — there's a brief period where the new capacity is being populated. The existing nodes handle load during this window; they just do so at higher per-node concurrency until migration completes.

Memory vs. Redis round-trips. The consistent hashing approach keeps hot state in process memory, which delivers excellent latency at the cost of making scaling events slightly more complex. This is the right tradeoff for live streaming, where message latency directly affects the perceived quality of the interactive experience.

Frequently Asked Questions

Q: Why is a dedicated chatroom infrastructure necessary for live streaming?
A: Standard messaging backends often struggle with the geometric scaling of chatrooms containing hundreds of thousands of concurrent users. Nexconn's Open Channel infrastructure uses a three-layer architecture with consistent hashing to isolate administrative logic from message distribution, ensuring that your chat panel remains performant even during massive traffic spikes.

Q: How does Nexconn handle message distribution at the scale of one million concurrent users?
A: Nexconn solves high-throughput distribution by sharding users across message service nodes using consistent hashing. This allows the system to parallelize delivery, meaning that one million deliveries happen concurrently across nodes rather than hitting a single server bottleneck.

Q: What is the benefit of Nexconn's notification-pull architecture over a standard push model?
A: A pure push model can overwhelm both the server and the client during traffic bursts. Nexconn’s notification-pull model sends a lightweight signal to clients, allowing them to pull message batches only when they are ready. Combined with randomized micro-jitter, this effectively prevents "thundering-herd" pull storms.

Q: How does Nexconn ensure low latency during large-scale broadcasts?
A: Nexconn maintains sub-120ms latency by utilizing a dedicated SD-CAN and distributing user state across localized message service nodes. By keeping state in-process memory rather than forcing constant round-trips to an external database, Nexconn minimizes the per-message latency impact.

Q: Can Nexconn's architecture be customized for specific community governance needs?
A: Yes. Nexconn's chat infrastructure includes native business logic hooks, such as user allowlists, granular role-based permissions, and support for Discord-like sub-channel hierarchies. This allows operators to enforce community governance at the infrastructure level rather than building expensive, custom middleware.