![](\"https://centrifugal.dev/assets/images/blog_redis_scaling_real_time_system-745341dc1da429728581d3aaf0093587.jpg\")
A lot of real-time systems spend most of their time doing nothing. Think about a notifications service, a presence dashboard, a \"new message\" badge — the connections sit open for hours, and messages arrive rarely. The interesting scaling question for such systems is not throughput, but how cheaply a single node can hold a large pool of connections that are mostly idle.
\nAn idle connection still isn't free: even with no application traffic, every connection costs a bit of CPU (Centrifugo runs periodic per-connection work — an application-level ping, presence refresh, subscription checks) and a bit of memory (goroutines, buffers, protocol state). Multiply that by hundreds of thousands and the constant, do-nothing overhead becomes the thing that decides how many connections fit on a node.
\nThis post measures that overhead and shows how five Centrifugo PRO options reduce it. One cuts CPU, the other four cut memory — and almost every byte they save turns out to come from a specific piece of the connection: a stack or a buffer.
\nWe hold 200,000 real WebSocket connections open against a single Centrifugo node and leave them idle. \"Idle\" here means the connections do nothing but stay alive — the server sends its application-level ping every 25 seconds (the default), the clients answer it, and that is the only traffic. Then we measure the server process: CPU (in cores) and resident memory (RSS).
\nWe measure the Centrifugo process specifically — CPU as the process's own user + system time over a steady-state window well after the connection ramp, memory as its RSS after a forced GC — so the numbers are the server's own consumption, not the whole machine's.
\nOne thing RSS does not include, and which matters when you size a machine: the kernel keeps its own structures for every socket — the tcp_sock, an inode, a dentry, a file — and charges them to slab, not to your process. We measured about 4 KB per socket, and it shows up in neither ps, nor top's RSS column, nor Go's runtime memory stats. It is real memory and cgroup v2 does count it against a container limit, so add it to every per-connection figure below when planning capacity.
At these connection counts the server uses well under one core out of sixteen, so it's the per-connection overhead we're isolating, not saturation.
\nEverything below is the server's own cost — the number that decides how many connections a node holds.
\nWith default settings, holding 200,000 idle connections costs:
\n| Metric | Value | Per connection |
|---|---|---|
| CPU | ~0.94 cores | ~4.7 µs/s |
| Memory (RSS) | ~5.4 GB | ~27 KB |
| Goroutines | ~400,000 | 2 |
Two goroutines per connection: one reads from the socket, one owns writes to it. The CPU is almost entirely the machinery of the periodic per-connection timers — for 200k connections that is a lot of independent timers for the Go runtime to track and fire.
\nIt's worth splitting that ~27 KB, because it explains everything that follows:
\n| Component | Per connection |
|---|---|
| Goroutine stacks | ~12.3 KB |
| Heap (buffers, protocol state, client struct) | ~12.7 KB |
| Runtime overhead | ~2 KB |
Nearly half of an idle connection is goroutine stacks. Keep that number in mind: two of the memory options below are, underneath, different ways of making it smaller, and the other two go after the heap half.
\nbatch_periodic_eventsEach connection runs several periodic tasks — the keepalive ping, presence updates, subscription expiration checks. By default each is an independent timer. With many connections, the Go runtime spends real CPU maintaining all of them and waking a goroutine every time one fires.
\nbatch_periodic_events groups these per-connection events onto a shared timing wheel instead of scheduling each one independently. The periodic work still happens on schedule; it's just dispatched in batches rather than as hundreds of thousands of separate timer wakeups.
{
\"client\": {
\"batch_periodic_events\": true
}
}
The effect on the same 200,000 idle connections:
\n| Metric | Default | batch_periodic_events |
|---|---|---|
| CPU | ~0.94 cores | ~0.41 cores |
| Memory (RSS) | ~5.4 GB | ~5.4 GB |
That's over 2× less CPU to hold the same pool — down to ~2.0 µs/s per connection. Memory is unchanged to within a rounding error: this option is purely about how periodic timers are scheduled.
\nBy default every connection has a dedicated goroutine that owns writes to its socket and blocks waiting for something to send. For an idle connection that goroutine does nothing almost all the time — but it still exists, and a parked goroutine still costs a stack.
\nwrite_delay batches outgoing messages: instead of writing each message immediately, the connection waits a short delay to coalesce whatever arrives into a single write, which reduces system calls under load. write_with_timer changes how that batching waits — it uses a shared timer to trigger the flush instead of a dedicated per-connection writer goroutine. For this post the interesting part is the side effect at rest: idle connections no longer each hold a writer goroutine.
{
\"client\": {
\"write_delay\": \"100ms\",
\"write_with_timer\": true
}
}
| Metric | Default | write_with_timer |
|---|---|---|
| Goroutines | ~400,000 | ~200,000 |
| Memory (RSS) | ~27 KB/conn | ~22.4 KB/conn |
| Goroutine stacks | ~12.3 KB/conn | ~8.2 KB/conn |
One goroutine per connection instead of two, and the saving is exactly one 4 KB goroutine stack.
\nThe tradeoff to be aware of: write_delay adds latency equal to the delay (100 ms here) before a message is flushed, because the connection waits that long to batch. For notification-style workloads that is usually invisible; for latency-sensitive request/reply it may not be what you want. Tune the delay, or leave it off, accordingly.
That leaves one goroutine per connection — the one reading the socket. It can't be removed the same way; something has to wait for the client's next message. But it turns out most of what it costs isn't the goroutine. It's the size of its stack.
\nGo grows a goroutine's stack on demand, doubling it whenever a call chain doesn't fit, and — this is the part that matters — a long-lived goroutine never really gives that space back. So the stack grows to the deepest call the goroutine has ever made, and stays there.
\nA connection's read loop reads a frame and then processes the command inline: the event handler, the broker, the broadcast fan-out. That is a deep call chain, and it only has to happen once — even just the initial connect command — for the read goroutine to grow to 8 KB and hold it for the entire life of the connection, however idle it then becomes.
process_commands_off_read_loop moves command processing to a short-lived goroutine that exits when the frame is done. The read loop itself stays shallow. The deep stack still gets used, but it goes back to the Go runtime's stack pool and is reused by the next connection that needs it, instead of being pinned to a connection that spends its life waiting.
{
\"websocket\": {
\"process_commands_off_read_loop\": true
}
}
| Metric | Default | process_commands_off_read_loop |
|---|---|---|
| Goroutines | ~400,000 | ~400,000 (unchanged) |
| Memory (RSS) | ~27 KB/conn | ~23 KB/conn |
| Goroutine stacks | ~12.3 KB/conn | ~8.2 KB/conn |
Same saving as removing the writer goroutine — around 4 KB per connection — but by a completely different route: the goroutine count doesn't change at all, its stack just halves from 8 KB to 4 KB. The two options are independent, and they stack.
\nThe tradeoff: handing each frame to another goroutine costs a goroutine start and two scheduler handoffs. For a connection that is genuinely busy, that shows up as a small increase in per-command latency. For a large pool of mostly-idle connections — the case this whole post is about — you are trading an operation that almost never happens for memory you pay for constantly.
\nuse_write_buffer_poolThe last piece isn't a goroutine at all. Every WebSocket connection holds a write buffer — 4 KB by default — whether or not it is writing anything. use_write_buffer_pool makes connections share a pool of write buffers instead, so a buffer is held only while a write is actually in flight.
{
\"websocket\": {
\"use_write_buffer_pool\": true
}
}
| Metric | Default | use_write_buffer_pool |
|---|---|---|
| Memory (RSS) | ~27 KB/conn | ~24.0 KB/conn |
| Heap | ~12.7 KB/conn | ~8.6 KB/conn |
Another ~4 KB per connection, this time out of the heap rather than the stacks. Under sustained write load the pool sees more contention than a per-connection buffer, so this option is meant for pools that are idle most of the time.
\nread_buffer_sizeWith the write buffer pooled, a heap profile of the tuned server shows that the\nWebSocket read buffer is now 47% of everything left. Each\nconnection holds one, and by default it takes the 4 KB buffer that net/http\nallocated for the request it was upgraded from.
Centrifugo lets you size it. A client command is small — a subscribe, a publish,\na pong — so 4 KB is far more than a mostly-idle connection needs.
\n{
\"websocket\": {
\"read_buffer_size\": 1024
}
}
| Metric | All previous options | + read_buffer_size: 1024 |
|---|---|---|
| CPU | 0.46 cores | 0.45 cores |
| Memory (RSS) | 15.3 KB/conn | 10.9 KB/conn |
| Heap | 8.3 KB/conn | 5.1 KB/conn |
That's another 4.4 KB per connection, and CPU does not move — the largest\nsingle memory saving in this post, and the cheapest.
\nDropping to 512 bytes saves only another 0.5 KB, so 1024 is the sweet spot. The\ntradeoff is that a client sending messages larger than the buffer needs more\nread syscalls to pull them in — irrelevant for command traffic, worth thinking\nabout if your clients push large payloads.
\nThe five options target different costs, so they stack cleanly:
\n| Configuration | CPU | RSS/conn | Stacks/conn | Heap/conn |
|---|---|---|---|---|
| Default | 0.94 cores | 27.0 KB | 12.3 KB | 12.7 KB |
batch_periodic_events | 0.41 cores | 26.9 KB | 12.3 KB | 12.6 KB |
write_with_timer | 0.98 cores | 22.4 KB | 8.2 KB | 12.3 KB |
process_commands_off_read_loop | 0.96 cores | 22.9 KB | 8.2 KB | 12.8 KB |
use_write_buffer_pool | 0.92 cores | 24.0 KB | 12.3 KB | 8.6 KB |
| First four together | 0.46 cores | 15.3 KB | 4.2 KB | 8.3 KB |
All five (+ read_buffer_size: 1024) | 0.45 cores | 10.9 KB | 4.2 KB | 5.1 KB |
With everything enabled, one node holds 200,000 idle connections for ~0.45 cores and ~2.2 GB — 52% less CPU and 60% less memory than the default. Per connection that's roughly 2.2 µs/s of CPU and ~11 KB of RAM.
\nLook at where the memory went. The default 27 KB was 12.3 KB of stacks and 12.7 KB of heap; tuned, those are 4.2 KB and 5.1 KB. The four memory options each removed roughly 4 KB — one goroutine stack, one goroutine's worth of stack, one write buffer, and most of a read buffer.
\nA note on CPU: the memory options are not entirely free. The first three individually cost between 2% and 7% CPU and together around 12%, while read_buffer_size costs nothing measurable. batch_periodic_events more than pays for all of it, which is why the combined figure is still less than half the default — but if you enable only the memory options, expect CPU to tick up slightly.
These options shine for the workload we benchmarked: large pools of mostly-idle connections. Notifications, presence, live badges, dashboards that update occasionally — anywhere the connection count is high and the message rate per connection is low.
\nThey matter less for high-throughput workloads. If every connection is receiving a steady stream of messages, the cost moves from the idle overhead we optimized here into the actual message fan-out and socket writes, which is a different benchmark. (write_delay still helps there — batching writes cuts system calls under load — but that's a throughput story, not the idle-pool story of this post.)
A couple of practical notes:
\nbatch_periodic_events is currently marked experimental. It's a scheduling change with no effect on delivery semantics, but the flag reflects that it's a newer option.write_with_timer only applies when write_delay is set — the timer-based flush is meaningless without a delay to flush against.process_commands_off_read_loop and use_write_buffer_pool are WebSocket-transport options, so they live under websocket rather than client. They do not affect SSE or HTTP-streaming connections, which don't hold a persistent read loop in the same way.Everything above is measured at 200,000, which is where the full comparison is\npractical to run. But once the tuned configuration was in hand, the obvious\nquestion was how far the same machine goes.
\nWe pointed the load generator at the same 16 vCPU / 30 GB node with every option\nfrom this post enabled, and asked it for 1,000,000 connections:
\n| Metric | Value | Per connection |
|---|---|---|
| Connections | 1,000,000 | |
| CPU | ~2.33 cores | ~2.3 µs/s |
| Memory (RSS) | ~11.3 GB | ~11.6 KB |
| Goroutines | 1,000,163 | 1.00 |
| — of which stacks | ~4.1 GB | ~4.1 KB |
| — of which heap | ~5.2 GB | ~5.2 KB |
Add the kernel's own per-socket structures — about 4 KB each, so roughly 4.2 GB\nfor a million sockets — and the true cost of the node is closer to 15.5 GB,\nstill only half the machine.
\nTwo and a third cores and 11 GB of process memory, on one node, for a million\nidle WebSocket connections. The per-connection cost is only slightly above the 200k figure —\n~11.6 KB against ~10.9 KB — so the tuning holds its shape at five times the\nscale rather than degrading.
\nTo be clear about what this run is: a single run of one configuration, not the\ntwo-round matrix the rest of the post is built on, and the connections are\nidle — they connect, answer pings, and nothing else. Real traffic adds its own\nbudget on top.
\nAnd one number that is not in the table, because we could not measure it: the\ndefault configuration simply cannot hold a million connections on this machine. At\nthe ~27 KB per connection we measured at 200k, a million of them would need\naround 27 GB of process memory on a 30 GB box — before the kernel's own\nper-socket structures, which add roughly 4 KB per socket on top. The tuned\nconfiguration does it in 11.3 GB and leaves half the machine idle.
\nAn idle connection costs CPU (periodic per-connection timers) and memory — and that memory is mostly goroutine stacks and buffers held by connections that are doing nothing. Centrifugo PRO gives you a knob for each:
\nclient.batch_periodic_events batches the periodic timers — over 2× less CPU.client.write_delay + write_with_timer drops the per-connection writer goroutine — ~4 KB less per connection.websocket.process_commands_off_read_loop halves the read goroutine's stack — another ~4 KB.websocket.use_write_buffer_pool shares write buffers between connections — another ~4 KB.websocket.read_buffer_size sizes the read buffer to what commands actually need — another ~4.4 KB, the largest saving of them all.The split between the client and websocket sections is not cosmetic — it is\nalso the split between what helps everywhere and what does not.
The two client options act on the connection object itself, which every\ntransport builds the same way, so batch_periodic_events and write_delay +\nwrite_with_timer help SSE and HTTP-streaming connections too: the periodic\ntimers and the per-connection writer goroutine exist regardless of how the\nconnection was established.
The three websocket options do not. Two of them size buffers that belong to\nthe WebSocket connection, and process_commands_off_read_loop only means\nanything where there is a long-lived read loop to keep shallow — SSE and\nHTTP-streaming receive client commands over ordinary HTTP requests, whose\ngoroutines are released when the request ends, so there is no persistent stack\nto shrink.
So if your fleet is not WebSocket, the CPU win and one of the four memory wins\nstill apply.
\nTogether they roughly halve the CPU and cut the memory of a large idle connection pool by 60% — enough to hold a million idle connections on a single 16-core node with half its RAM to spare. If your workload is a lot of connections that mostly wait, they're worth turning on.
", "url": "https://centrifugal.dev/blog/2026/08/17/tuning-idle-connections", "title": "Tuning Centrifugo PRO for large number of idle WebSocket connections", "summary": "An idle WebSocket connection costs CPU and memory even when no messages flow. The post starts with 200k idle connections on a single node, cuts the overhead with five Centrifugo PRO options, then holds a million idle connections on the same node.", "date_modified": "2026-08-17T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "websocket", "performance" ] }, { "id": "https://centrifugal.dev/blog/2026/07/27/app-owned-state-stream-subscriptions", "content_html": "Most real-time features start from data that already exists somewhere — orders in an orders table, notifications, a document, a ticket queue. To show it live, the browser needs two things: the current state, and every change that happens after. In a typical app the state comes from an HTTP endpoint and the changes come from a WebSocket subscription.
Each part is easy on its own. The hard part is joining them correctly, and it turns out to be hard in the same way in every application that keeps state in its own database.
\nThe natural approach is to load the state first and subscribe after. Three problems hide in this sequence.
\nThe first is a gap in time. Between the moment the HTTP response was produced and the moment the subscription becomes active, updates keep happening. They are not in the loaded snapshot, and the subscription started too late to catch them, so they are simply lost. The component shows stale data until the user reloads the page.
\nThe second one shows up later, on reconnects. After a network drop the SDK asks Centrifugo to replay the publications the client missed. Usually it can. But sometimes it can't — the client was offline longer than the history retention, or the stream epoch changed. The subscription then reports recovered: false and the application has to reload state from the backend. While that reload is in flight, new publications keep arriving — the same race as on the first load, except now it can happen on any reconnect.
The third is ordering between the two sources. If a publication arrives while the state fetch is still running, it has to be applied on top of the fetch result, not before it — otherwise a fresh value gets overwritten by an older one.
\nNone of this is specific to Centrifugo: any system that combines \"fetch a snapshot\" with \"subscribe to a feed\" runs into the same three problems. Proper real-time document state synchronization showed how to solve them by hand — buffer publications during the fetch, track versions, re-sync when continuity is lost. It works, but it's a fair amount of code to repeat in every client.
\nStream subscriptions now have this logic built in.
\ngetState is an option on a stream subscription. The SDK calls it before subscribing. Inside the callback the application loads its state and returns the stream position that state corresponds to:
const sub = client.newSubscription('orders:user_42', {
getState: async () => {
// 1. Capture the stream position FIRST.
const pos = await api.getStreamPosition('orders:user_42');
// 2. Then load and render your data.
renderOrders(await api.getOrders(42));
// 3. Return the position — the SDK subscribes from exactly here.
return { offset: pos.offset, epoch: pos.epoch };
},
});
sub.on('publication', (ctx) => {
// Incremental updates: catch-up first, then live.
applyOrderUpdate(ctx.data);
});
sub.subscribe();
This is the entire integration. The application answers one question — what is the current state, and at which stream position was it taken — and the SDK takes care of everything else.
\n getState() subscribe(recover from pos)
│ │
▼ ▼
┌─────────┐ pos ┌───────────────────────────┐
│ App API │ ─────────────► │ catch-up publications │──► live
│ + DB │ state │ (offset > pos.offset) │
└─────────┘ └───────────────────────────┘
▲ │
│ recovery impossible — reload from scratch │
└─────────────────────────────────────────────────────┘
It's worth walking through the behaviour case by case, because it defines how much the application still has to handle on its own.
\nOn the first subscribe, the SDK runs getState and subscribes with recovery from the returned position. Everything published after that position arrives through the normal publication handler, so catch-up publications and live ones look identical to the application.
On a normal reconnect, getState is not called. The SDK already holds a valid position, so it lets the server replay the missed publications from channel history. This is exactly what recovery exists for: when thousands of clients reconnect at once, the broker absorbs the load instead of the application database.
When recovery turns out impossible, getState runs again — and this is the part that replaces the manual re-sync code. A subscription with getState asks the server to reject a subscribe it cannot recover, instead of accepting it with recovered: false. The server responds with the unrecoverable position error, the SDK drops its saved position, and the next subscribe attempt calls getState again: fresh state, fresh position, recovery from there. At no point is the subscription live while the state under it is stale.
And if getState itself fails — a network error, a database timeout — the SDK emits a subscription error and retries with the usual backoff, so a failed state load never leaves a component half-subscribed.
There is one rule the application has to get right inside the callback: read the stream position before reading the data.
\nIn this order the position becomes a lower bound. Anything committed between the two reads shows up twice — once in the loaded data, and once more as a catch-up publication. Seeing an update twice is harmless, missing one is not, and reading the data first would produce exactly the kind of gap this whole mechanism exists to close.
\nSince duplicates are possible, the application must be able to apply the same update twice. Most update shapes already can:
\nshipped\", \"price is 42.17\". Applying it twice changes nothing.id, keep the newer updated_at. Last write wins.And when the data source is a single relational database, the overlap can be removed entirely: wrap both reads in one REPEATABLE READ transaction. Both statements then see the same MVCC snapshot, the returned position is the exact watermark of the loaded data, and catch-up delivers only what was committed strictly after it. Nothing left to reconcile, at the cost of a transaction on the read path.
getState must return an offset and an epoch for the channel. There are two ways to get them.
With any broker, the server API works: a history call without a limit returns just the current stream position, without publications:
curl -X POST http://localhost:8000/api/history \\
-H \"X-API-Key: <key>\" \\
-d '{\"channel\": \"orders:user_42\"}'
{\"result\": {\"offset\": 174, \"epoch\": \"xcf4\"}}
The application's own endpoint calls this, then reads its data, then returns both to the client. Clients never talk to the Centrifugo API directly.
\nWith the PostgreSQL stream broker, the position lives in the same database as the data, so both reads happen in one place — and, if needed, in one transaction:
\nSELECT * FROM cf_stream_top_position('orders:user_42');
SELECT id, status, updated_at FROM orders WHERE user_id = 42;
This is what makes the PostgreSQL broker such a natural fit for app-owned state: the position and the rows share a transaction boundary, so the REPEATABLE READ trick above becomes a two-line change. The write side has the same property — cf_stream_publish commits the publication together with the row it announces, which was the subject of Transactional publishing for stream subscriptions with PostgreSQL.
getState builds on standard history and recovery, so the channel namespace needs history and recovery enabled:
{
\"channel\": {
\"namespaces\": [
{
\"name\": \"orders\",
\"history_size\": 100,
\"history_ttl\": \"300s\",
\"force_recovery\": true
}
]
}
}
Note that the history window only has to cover the reconnect gap, not the lifetime of the data. Anything longer than that is served by getState reloading from the application database — which is the right split: channel history is a bounded cache, the application database is the source of truth.
Alternatively use allow_recovery and let clients opt in per subscription with the recoverable option.
The rules above are short, so here are two examples showing how they play out in real systems. Both use the PostgreSQL stream broker, which makes the write side transactional too — but the read side works the same with any broker.
\nThe first one is a service that consumes a Kafka topic and maintains aggregated views in PostgreSQL — say, a price board built from a market-data topic. The browser needs the current aggregate, then live updates.
\nThe obvious wiring is to point Centrifugo at the same Kafka topic and let it fan events out to subscribers in parallel with the aggregator. That works, but it leaves two unrelated offset spaces to bridge: the snapshot row stores a Kafka offset, the live subscription speaks Centrifugo offsets, and the client has to subscribe with a recent stream position and throw away everything older than the snapshot's Kafka offset. This bridging logic is awkward and easy to get subtly wrong.
\nThere is a simpler wiring: make the aggregator the publisher. For each Kafka batch it processes, it does both things in one PostgreSQL transaction — updates the snapshot and calls cf_stream_publish(...) for the new events. Now the stored aggregate and the change stream can never disagree about what has been observed, and the snapshot row stays minimal: just the aggregate, no offset bookkeeping.
─── Write path (single PG txn ties snapshot + publish) ───────────
Kafka topic
│ batch
▼
┌────────────┐
│ Aggregator │
└─────┬──────┘
│ BEGIN
│ UPDATE snapshot SET aggregate = ...
│ cf_stream_publish(p_channel := ch, p_data := evt)
│ COMMIT ← both land atomically
▼
┌──────────────────────────────┐
│ PostgreSQL │
│ ┌──────────┐ ┌──────────┐ │
│ │ snapshot │ │cf_stream │ │
│ │ row │ │ outbox │ │
│ └──────────┘ └────┬─────┘ │
└────────────────────┬┴────────┘
│ LISTEN/NOTIFY + poll
▼
┌──────────────┐
│ Centrifugo │
│ (PG broker) │
└──────────────┘
─── Read path (position first; catch-up applied idempotently) ────
Browser ──1. GET /state──► App server
│ pos = cf_stream_top_position(ch)
│ snap = SELECT aggregate FROM ...
Browser ◄──── (snap, pos) ─────┘
Browser ──2. subscribe(ch, since=pos)──► Centrifugo
Browser ◄─── catch-up from pos → live ──┘
(events committed between the two reads
arrive here; idempotent apply reconciles)
The read path is the recipe from the previous sections: cf_stream_top_position first, then the snapshot, then return the captured position. Events committed between the two reads arrive as catch-up on top of the snapshot. A price board reconciles this on its own, since each event carries an absolute price rather than a delta. If the events were deltas, the REPEATABLE READ variant would remove the overlap along with the need for dedup.
The same shape fits any upstream feed being turned into stored views — Kafka, NATS, CDC, a polling worker. One process owns both the stored aggregate and the change stream, so there is no cross-system offset bridge to maintain.
\nThe second example is even more common: one shared table, many independent consumers. The kitchen-orders demo shows it well — a single orders table across all restaurants, one channel per restaurant (kitchen:{restaurant_id}), and each kitchen display subscribed to its own channel. The write side follows one rule: every code path that changes a row for restaurant X publishes on kitchen:X in the same transaction.
The read side behind getState is the position-first recipe, with data filtered by restaurant:
SELECT * FROM cf_stream_top_position('kitchen:42');
SELECT id, status, items, updated_at
FROM orders
WHERE restaurant_id = 42
AND status IN ('received','preparing','ready');
Events carry order_id and updated_at, so the client applies them as upserts with last-write-wins — the keyed-upsert case from the list above, which makes catch-up replay harmless. There are no offsets stored in the application schema and no dedup bookkeeping: the position is captured at read time and handed to the SDK.
The same client code serves every restaurant — only the channel name and the WHERE clause change. That's what lets this shape scale to thousands of tenants without any per-tenant machinery.
The pattern assumes the data has a home in the application's schema. Plenty of real-time data does not: cursors that live for a few seconds, presence sets, lobby members, device telemetry, feature flags. Building a table, a snapshot endpoint and a change feed for each of those is a lot of machinery for state that is disposable by nature.
\nFor such data the broker can act as the store instead — see map subscriptions for keyed collections, and cache recovery mode when a channel only ever needs its latest value.
\nA simple rule of thumb: if a SELECT in your own database can already answer \"what is the current state\", then getState on a stream subscription is the smaller design. If it can't, look at the state-carrying subscription types.
The getState callback is available in Centrifugo v6.8.0+ and is supported by all bidirectional SDKs — JavaScript, Dart, Swift, Go, Java, Python.
See the Subscription getState API reference for the option details, and the pg_stream_broker example for a runnable app built on this pattern.
", "url": "https://centrifugal.dev/blog/2026/07/27/app-owned-state-stream-subscriptions", "title": "App-owned state with stream subscriptions", "summary": "Describing the getState callback on stream subscriptions — capture the stream position first, load the data, and let the SDK handle every reconnect after that. With two worked examples — a Kafka aggregator and per-tenant kitchen orders.", "date_modified": "2026-07-27T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "websocket", "docsync" ] }, { "id": "https://centrifugal.dev/blog/2026/06/29/scaling-redis-pub-sub", "content_html": "Redis Pub/Sub is a popular choice for passing messages between nodes in real-time messaging systems. It lets a system run many nodes — each holding many real-time client connections — and deliver each message to the nodes that have interested subscribers.
\n
At small scale things are quite simple. But at millions of channels and hundreds of subscriber nodes — the scale some Centrifugo users run at — Pub/Sub stops being a simple part of the design: a single Redis instance is limited by one CPU core, and switching to Redis Cluster can make throughput worse. This post walks through those gotchas, and the techniques that get from a single instance's ceiling to millions of messages per second — across isolated Redis shards and specifically in a Redis Cluster.
\n\nThis post talks about Redis, but everything here applies equally to Valkey too.
For a general-purpose real-time messaging server, every deployment looks different, so the design has to handle many cases. Two facts about real-time messaging specifics shaped the decisions in this post:
\nRedis Pub/Sub is intentionally simple: a client subscribes to a named channel, and a publish to that channel reaches whoever is subscribed at that exact moment. Nothing is stored and nothing is retried, so a subscriber that wasn't connected when a message went out never sees it — delivery is at most once.
\n\nRedis runs every command (in this case PUBLISH and SUBSCRIBE) on a single thread sequentially. Modern Redis and Valkey can spread network I/O across extra threads, which lifts throughput. Also, an application may issue subscriptions to Redis replicas to spread the load a bit. However, command execution stays on the Redis master and remains serialized, so one instance still has a performance ceiling you can hit.
\nWhatever Redis setup you run, your app has to keep up with Redis first — otherwise it's the first bottleneck, before Redis itself is even reached. With Redis Pub/Sub, each application node — also called a subscriber node here — does two things: it publishes messages and subscribes to receive them.
\nCentrifugo publishes over a pipelined connection instead of the connection-pool approach, where each command takes its own network round trip. The rueidis client gathers commands issued close together and writes them as a single batch, with a small flush delay (MaxFlushDelay, about 100µs) marking a batch boundary. One round trip then carries many publishes, so the publisher feeds Redis instead of stalling on the wire.
Pipelining lifts the overall throughput and surprisingly cuts CPU usage on both the client and Redis sides due to reduced READ/WRITE syscalls — these gains were shown in detail before in Improving Centrifugo Redis Engine throughput and allocation efficiency with Rueidis Go library.
\nThe SUBSCRIBE command supports subscribing to many channels at once. When many clients reconnect at once or application nodes resubscribe after a network glitch, a few large commands with batched channels are far more efficient across the client, protocol, and Redis layers.
Redis keeps a buffer for each Pub/Sub connection, and if the reader doesn't drain it fast enough, Redis drops the connection. So the application has to drain the socket as fast as it can and delegate processing to a dedicated pool of workers. Because Pub/Sub delivery is at most once, the workers have some freedom in how they handle overload — including dropping messages at the application level.
\nIt's worth tracking an application metric for the end-to-end lag of a message sent through Pub/Sub — record the publication timestamp on the publisher side and subtract it from the time the message is received on the subscriber side. In most cases, workers falling behind is a signal that it's time to scale the application nodes.
\nTo keep a channel's messages in order, pick the worker by a hash of the channel name: every message for a channel goes to the same worker, while different channels spread across all the available cores. Also, using a few pipelined connections instead of one can further raise read throughput.
\n\nAdding more application nodes helps absorb higher throughput, but keeping each node fast still matters. Every node opens its own connections to Redis, and at scale the total connection count becomes a concern of its own — more on that later.
\nA single Redis instance is very fast, and I/O threading lets it use several cores for network work — but it still runs every command on one thread, and once that thread saturates there's nothing Redis can do.
\nIn a Centrifugo benchmark on Hetzner, on a machine with dedicated vCPU, a single Redis instance handled up to about 650,000 messages per second (not just published, but all delivered to active subscribers, each message 256 bytes). After that, end-to-end latency started to grow fast — a signal of saturation. This throughput should be enough for most use cases. But to go further you have to stop using a single Redis instance.
\nThe simplest solution is to spread the Pub/Sub load across several independent Redis instances. The application decides which Redis instance handles a given channel by hashing the channel's name, for example with a Jump consistent hash or any other. Consistent hashing here may help to keep most of the data on the same Redis instances after adding a Redis node and re-sharding. In Centrifugo's case, we keep channel history in Redis in addition to Pub/Sub — the protocol is designed to tolerate the loss of it and provide a signal to real-time clients, so the loss during re-sharding is tolerable.
\nIn such a setup, a publisher and a subscriber should use the same hash function over the channel name, so they always land on the same instance without having to talk to each other, and the instances don't even know about one another — so each instance you add takes a share of the load.
\n\nIn our benchmarks Pub/Sub throughput increased linearly as more Redis instances were added — each one took an even share of the load, and latency stayed low. So 650k messages per second became around 5M messages per second with 8 Redis instances. The instances share nothing, so the total capacity is just one instance's limit times the number of instances. Connection count naturally grows with the number of shards.
\nThe number of connections per subscriber node is multiplied by the number of Redis shards. For example, for 8 separate Redis instances and 24 subscriber nodes it's 24 x 8 x (N pipeline conns).
\nClient-side sharding works, but requires manual configuration. The list of Redis instances lives in the server's own configuration, so adding capacity isn't just starting another Redis — every node has to reload the new list.
\nThe obvious next step is to switch to Redis Cluster, which manages the group of nodes for you. But it's not that simple.
\nA single Redis holds everything in one process, while Redis Cluster spreads the data across several nodes. It does this by splitting all possible keys into 16,384 buckets called hash slots: every key belongs to exactly one of them, chosen by CRC16(key) % 16384, and each node owns a range of slots. To reach a key, the client hashes it and goes straight to the node that owns that slot.
There's also a handy trick called a hash tag. If a key contains braces, Redis only hashes the part inside the {...}, which lets you force a group of keys into the same slot — and so onto the same node — whenever you want them to stay together.
Because keys are spread across nodes this way, ordinary operations scale cleanly. A GET or SET goes only to the node that owns the key, so adding nodes spreads both the data and the load, and throughput grows almost linearly — this is the main benefit of Redis Cluster. Pub/Sub is the exception.
Ordinary Pub/Sub in Redis Cluster sends every published message to every node — even the nodes where nobody is subscribed to that channel. It has no real choice: a subscriber could be attached to any node, so the only way to be sure they get the message is to copy it everywhere. That sounds harmless until you see the cost — the more nodes you add, the worse it gets, because each one has to carry every message, including all the ones it has no subscriber for.
\n\nThere's a second, less visible problem with this. With classic Pub/Sub you can't choose which node handles your subscriptions — they all go over a single connection to whichever node it happens to point at for the first subscribe operation. So one node can end up tracking every subscription while the rest sit nearly idle — a subscription imbalance.
\nIn our benchmarks, a classic Redis Cluster behaved just as the theory above says. Under load in a 3-node Redis Cluster, the delivery latency grew from milliseconds to whole seconds long before the 650k messages per second a single instance reached, somewhere around 400k messages per second. And it got even worse as nodes were added.
\nRedis 7 introduced sharded Pub/Sub (SSUBSCRIBE / SPUBLISH commands) to fix exactly this. Within these new commands a Pub/Sub channel respects hash slots and behaves like a normal key: a message goes only to the Redis node that owns the channel's slot, instead of being copied to every node — which solves the throughput problem. The price is that Pub/Sub now has to be coordinated by the Redis client driver: each subscribe and publish must go to the node that owns the slot, instead of one shared connection to any Redis Cluster node.
This visualization shows the benefit of sharded Pub/Sub and why classic Pub/Sub fails to increase Pub/Sub throughput:
\nFor some applications, moving to sharded Pub/Sub is easy — just call the new methods in the Redis client driver. For Centrifugo's workloads it was not that simple, here is why.
\nWith classic Pub/Sub, subscribing was easy: take a Pub/Sub connection and send SUBSCRIBE commands over it, without caring which Redis node it points to. With sharded Pub/Sub, the connection returned by the rueidis driver points to the node that owns the slot of the first channel subscribed to. SSUBSCRIBE also supports a batch of channels, but all must belong to the same hash slot.
Subscribing to a second channel can't reuse the existing Pub/Sub connection — it needs another one. With only a handful of channels that's fine: open a few sharded Pub/Sub connections and you're done. But with millions of channels that approach no longer works — a million connections to Redis isn't something anyone would attempt. So there's a problem with connection count, and also it's not obvious how to keep the subscribe batching that helped to keep the system effective.
\nThe way around it is to stop treating channels individually and group them on application level. Each channel is hashed into a small, fixed number (N) of partitions, far smaller than the channel count. And the number of partitions must be larger than the Redis Cluster size for better distribution. By default, Centrifugo uses 128. Every channel in a partition gets the same hash tag, so they all land in the same slot, on the same Redis node. A channel key ends up shaped like {<partition>}.<channel>:
SSUBSCRIBE news sports # rejected if they hash to different slots
SSUBSCRIBE {42}.news {42}.sports # shared {42} tag → one slot → one connection, batched
A single partition then uses a single connection: when the first channel in a partition is subscribed, the client opens one connection to that partition's node, and every other channel in the partition reuses it. Because those channels share a slot, their subscriptions batch and pipeline on that connection, the way classic Pub/Sub batched.
\nAnd because the partitions are spread across the cluster, the connections spread with them instead of piling onto one node. So instead of a connection per channel there's one per partition — N per node — and subscribing is back to cheap, batched commands over a handful of steady connections per node, even as channels come and go. That settles the count on a single node; multiplied across a large fleet the cluster-wide total is another matter — more on that below.
\n\nThe partition scheme skips over one important detail: what to use as each partition's tag. The obvious answer is the partition's own index — {0}, {1}, {2}, … {N} — the natural starting point. It works, but it has a hidden problem. Run those short strings through CRC16 and the slots they hash to aren't evenly spread, so some Redis nodes end up with many more partitions than others.
Take, for example, a 6-node cluster with N = 128 partitions: with naive {0}…{127} tags the partitions land [27, 15, 22, 23, 16, 25] across the six nodes — the busiest node gets nearly twice as many as the lightest, and the gap only grows on bigger clusters. That skew has a real cost: Pub/Sub throughput is capped by the busiest node, so it saturates first while the lightest still has headroom — you pay for all six nodes but leave capacity unused, and that's the opposite of what you add nodes for.
So the tags need to be chosen more carefully. For a given number of partitions, we've implemented a small simulated-annealing search script — it finds a set of short tag strings whose CRC16 slots stay balanced across every cluster size from one node up to the partition count, not just a few sizes picked in advance. It runs once and bakes the result into a static table of string precomputed tags Centrifugo can then use, so at runtime it's just a lookup in that table.
\nOn that same 6-node cluster the 128 partitions now land [21, 22, 21, 21, 22, 21] — near-even, instead of the uneven spread the naive tags gave. Move to 12 nodes in Redis Cluster, and the tags still hold — [11, 10, 11, 11, 10, 11, 11, 10, 11, 11, 10, 11], while naive tags would give [10, 17, 5, 10, 16, 6, 8, 15, 9, 7, 16, 9]. Past 128 nodes you'd have more nodes than partitions, so some would sit empty — but by then you'd raise the partition count.
Here is a visualization of that:
\nSharded Pub/Sub, combined with app-level partitions, solved the throughput problem — each message reaches a single node, and the cluster scales the way client-side sharding does. Redis nodes get balanced load. But a new problem shows up as the system grows: the connection count. There's a fixed number of partitions and many subscriber nodes, each opening its own connection per partition — and those two counts multiply together.
\nHere's the math. Under the partition scheme each node opens one subscribe connection per partition, so 128 partitions across 200 nodes works out to 200 × 128 = 25,600 connections into the cluster, and spread over a 6-node cluster that's about 4,267 of them landing on each Redis node. That's the point where you start hitting real limits — maxclients, ephemeral ports, file descriptors, the overhead Redis and the app carry for every connection. This comes from a real setup — a Centrifugo user running 20 million WebSocket connections on AWS.
Most of those connections are unnecessary. Each Redis node owns a whole range of slots, so 20-21 of the 128 partitions live on any given node — and a single connection to that node can subscribe to all of them. So it makes sense to group subscribe connections by Redis node.
\n\nWith the same 200 nodes and the same 6-node cluster, the count then drops to 200 × 6 = 1,200 connections in total, just 200 per Redis node instead of 4,267.
With this decision, the partition count can then be raised — 1024 or 4096 instead of 128 — without adding connections. A higher count spreads channels across more slots, for a finer, more even distribution on larger clusters.
\nConnections are under control now. But the partitions earn their keep a second time, in a different place — reconnect storms.
\nPartition grouping makes no difference during normal work: subscribing to one new channel is a single SSUBSCRIBE, no matter how channels are grouped. A network glitch or a failover, though, drops the connections, and across a large fleet many nodes resubscribe everything they hold at the same moment — that's when replaying all those subscriptions in as few commands as possible starts to count.
There's a catch in how a node sends those resubscribes, and it's easy to get wrong. To resubscribe quickly the node splits its channels across several workers that run at the same time — say 16 of them. The obvious way to split the work is to give each worker a share of the channels. But one SSUBSCRIBE can only carry channels from a single partition, and if each worker holds a mix of channels from every partition, each worker has to send a separate SSUBSCRIBE per partition. With 21 partitions and 16 workers that's 16 × 21 = 336 commands, each carrying a thin slice of a partition.
Split the work by partition instead — give each worker a few whole partitions to subscribe — and every partition is one SSUBSCRIBE again: 21 commands, not 336. The channels and the workers are the same — only the way the work is divided changes. On a real cluster this one change took a node's resubscribe from a few hundred commands down to a couple dozen.
Partitions are why that command count is 21 and not thousands in the first place. On a 6-node cluster with 128 partitions each Redis node owns about 21 partitions. Drop partitions and that node owns about 2,731 of the 16,384 slots instead. So when a node resubscribes, its connection to one Redis node sends about 21 SSUBSCRIBEs with partitions, or about 2,731 without — 130 times more. The count never goes past 16,384, and it never grows with the number of channels. (Each SSUBSCRIBE carries at most the batch size — 512 in Centrifugo's case — so once a partition holds more than that, both cases send about channels / 512 commands and the gap closes.) That's nothing on its own, but in a fleet-wide reconnect every subscriber node fires its commands at each Redis node at once — right when the cluster is busy recovering — so keeping the per-node count low matters most exactly then.
So partitions stay — not for the connection count any more, but to keep these big resubscribes cheap. Dropping them is a fair choice too: you get the finest, most even distribution, but each resubscribe costs more. That's the trade-off you control.
\nGrouping subscriptions by node isn't free — it adds complexity, mostly around tracking the cluster topology. This mechanism won't be available out of the box in most Redis drivers. Luckily the rueidis library Centrifugo uses to talk to Redis exposes enough low-level Redis Cluster knobs to build it.
\nBecause the application node decides which Redis node each subscription goes to, it has to track the cluster's layout itself. It keeps a small map — each partition to the Redis node that owns its slot — built from CLUSTER SLOTS and the client's node list. Each Redis node gets one connection carrying the partitions it owns. The cluster can reshuffle this at any time — a node joins, a node leaves, or slots move to rebalance — and the app has to notice each change and rewire.
There are two ways it knows about changes. A background loop re-runs CLUSTER SLOTS every 30 seconds (in Centrifugo's case) and rebuilds the map; if the node count has changed, or any partition now points at a different node, the layout has changed. But the poll isn't the only signal. When the cluster moves a slot off a node, Redis sends a sunsubscribe for the sharded channels held there on its own — the node announcing it no longer owns them. The subscriber node catches that signal on the subscribe connection and triggers a rebuild right away, so a removed node or a moved slot doesn't wait up to 30 seconds to take effect.
A rebuild recomputes the map and rewires the connections: open one to a Redis node that appeared, drop the one to a node that left, and resubscribe each partition on its new owner. Publishes need no help — a cluster-aware client already sends each SPUBLISH to whichever node owns the slot — so once the subscribe side re-points, publisher and subscriber meet on the new node again.
One catch: just after a node joins, the rueidis client may not have a connection to it yet. Until it does, the slots that new node owns don't map to any node the client knows about, so the partition-to-node map can't be completed. To push the client to update, the subscriber node issues a throwaway GET for a probe key to provoke a MOVED redirect — that makes rueidis re-read the cluster layout and pick up the new node in time for the next rebuild.
With node grouping in place, a Redis Cluster ends up working like client-side sharding: every Redis node is an independent single instance, owning its own slots and carrying no cross-node Pub/Sub traffic. Each node's ceiling is a single core — about 650k messages a second in these benchmarks — and throughput scaled linearly as cluster nodes were added, each one giving about another 650k, the same as adding client-side shards. Run side by side, the two were hard to tell apart: low latency, with the load split evenly and each node carrying its 1/N share.
\nHere are the two approaches compared:
\n| Client-side sharding | Node-grouped Redis Cluster | |
|---|---|---|
| Throughput | N × single-Redis | the same |
| Connections | nodes × Redis instances | nodes × Redis instances (similar) |
| Membership & failover | you build it yourself | the cluster handles it |
| Adding capacity | edit the instance list by hand, every node reloads | add a node, slots rebalance on their own |
| What you maintain | a Redis list in config | extra code for node-grouping |
Client-side sharding is simpler — it's a list of Redis addresses and a hash function, nothing more — and for a lot of deployments that's plenty, as long as you don't mind editing that list and resharding by hand as you grow. Redis Cluster is worth its extra complexity when you'd rather not do that by hand: it manages membership, failover, and resharding itself, and node grouping is what keeps its Pub/Sub scaling while it does.
\nA single Redis Pub/Sub instance scales well until one core saturates. Past that you need more Redis nodes, either through client-side sharding or Redis Cluster.
\nRedis Cluster, though, broadcasts every published message to all nodes by default — so adding nodes lowers throughput instead of raising it, and subscriptions can pile unevenly onto one node.
\nSharded Pub/Sub fixes that by turning the cluster into a set of independent Redis instances, just like client-side sharding — so capacity scales linearly as you add nodes, but without you having to manage the instance list yourself.
\nFor Centrifugo's specific situation — a large number of active channels, a high subscribe/unsubscribe rate, reconnect storms, and lots of subscriber nodes — a few more tricks were needed to make it actually work:
\nSSUBSCRIBEs per Redis node instead of hundreds.None of this is a recipe for endless scale. What it gives is a good state to grow from: the Pub/Sub layer stays balanced as the cluster gets larger, and the connection count no longer grows out of control. So you can move to a bigger Redis Cluster and run more subscriber nodes without the number of connections becoming the thing you hit first. Where the real ceiling lands after that depends on the rest of the system — the publishers, the application nodes, the traffic per channel — not on the Pub/Sub itself.
\nPub/Sub is not the only load an application generates. You usually also have plain GET/SET operations, which scale well in Redis Cluster without extra engineering. It's often worth running Pub/Sub and GET/SET on separate Redis setups, so each one can be tuned independently.
\nThank you for your attention!
", "url": "https://centrifugal.dev/blog/2026/06/29/scaling-redis-pub-sub", "title": "Scaling Redis Pub/Sub to Millions of Channels and Hundreds of Subscriber Nodes", "summary": "How Centrifugo scaled Redis Pub/Sub — talking to Redis efficiently, sharding across isolated Redis instances, and making Pub/Sub work on Redis Cluster: sharded Pub/Sub, slot balance, connection count, and efficient resubscribes.", "date_modified": "2026-06-29T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "redis", "pubsub", "scalability" ] }, { "id": "https://centrifugal.dev/blog/2026/05/25/pg-controller-multi-node", "content_html": "For most of Centrifugo's history, scaling past one node meant adding Redis. The new PostgreSQL controller makes Redis optional. Together with the stream broker and map broker shipped earlier in this release cycle, Centrifugo's full OSS messaging cluster now runs on the PostgreSQL only.
\nAvailable in Centrifugo v6.8.0+. The PostgreSQL controller is a recent addition — we're eager for feedback. Configuration keys, schema, and outbox internals may still adjust before they're considered stable.
A single Centrifugo node is self-contained — clients connect, the node tracks subscriptions in memory, publishes go straight to subscribers. Once a second node added, that breaks: a publish that arrives at node A needs to reach the clients connected to node B; subscribes, unsubscribes, and disconnects need to propagate; each node needs to know the live cluster topology. Centrifugo solves this with a Controller interface that distributes control messages between nodes.
Before v6.8.0, that interface had two implementations: Redis and NATS. NATS, however, is a Centrifugo PRO option. So the OSS reality was effectively Redis or single-node. Teams already running Redis for caching paid no extra operational cost. Teams running PostgreSQL as their primary store and nothing else had to provision and operate a Redis instance just to scale Centrifugo horizontally — even small deployments where PG could have handled the messaging traffic just fine.
\nThat gap is what the PostgreSQL controller closes.
\nThe controller uses the same outbox pattern that powers the PG stream broker and the PG map broker:
\nLISTEN/NOTIFY provides low-latency wakeup, so polling cadence stays low under quiet load while latency stays in the low single-digit milliseconds when traffic exists.The shape simply mirrors PostgreSQL brokers we described in recent blog posts.
\nThe controller carries cross-node operations Centrifugo needs to coordinate:
\nPublication delivery itself does not flow through the controller — that's the broker's job. Stream and map brokers each have their own outbox tables and their own delivery paths. The controller is purely the cross-node coordination layer.
\nWith the stream broker, the map broker, and the controller all on PostgreSQL, an OSS cluster can run with one infrastructure dependency for the entire messaging plane:
\n{
\"broker\": {
\"enabled\": true,
\"type\": \"postgres\",
\"postgres\": { \"dsn\": \"...\", \"use_notify\": true }
},
\"map_broker\": {
\"type\": \"postgres\",
\"postgres\": { \"dsn\": \"...\", \"use_notify\": true }
},
\"controller\": {
\"enabled\": true,
\"type\": \"postgres\",
\"postgres\": { \"dsn\": \"...\", \"use_notify\": true }
}
}
The same DSN can be reused across all three components, or split across separate PostgreSQL instances if you want to isolate the data plane from the control plane. All three create their own tables, partitions, and SQL functions on startup — there's no manual migration step.
\nThe PostgreSQL controller doesn't replace Redis or NATS for everyone. A rough way to decide:
\nuse_notify.For OSS deployments where Redis was added only to support multi-node Centrifugo, the PG controller is the cleaner option going forward. For deployments already running Redis as a cache or session store, there's no need to migrate off it — no functional reason to switch if Redis is already there for other purposes.
\nEnable the PostgreSQL controller alongside any existing broker configuration:
\n{
\"controller\": {
\"enabled\": true,
\"type\": \"postgres\",
\"postgres\": {
\"dsn\": \"postgres://user:pass@localhost:5432/app?sslmode=disable\",
\"use_notify\": true,
\"partition_retention_days\": 7
}
}
}
The controller creates its required tables, partitions, and SQL functions on startup. Bring up two or more Centrifugo nodes pointing at the same PostgreSQL, and they'll coordinate without further setup.
\nFor background on the outbox pattern shared with the brokers, see the PG stream broker post and the PG map broker post. The full configuration reference lives in the engines documentation.
\nA working three-node example lives in examples/v6/pg_cluster_demo. It boots a single PostgreSQL container, three local Centrifugo nodes pointing at the same DSN, and a static page that exercises all three components at once:
![](\"https://centrifugal.dev/img/demo_pg_only.jpg\")
\n/api/info and lists every node — that list is built from the heartbeats flowing through the PG controller.map_clients map subscription whose state lives in PostgreSQL via the PG map broker, so every tab sees the same presence rows regardless of which node served it. We generally don't recommend PostgreSQL for presence — it's lightweight ephemeral data that fits Redis better. But for use cases with a reasonable number of concurrent clients, it works fine.LISTEN/NOTIFY wakeup.Each browser tab connects to a specific node via ?n=1|2|3, which makes the cross-node fan-out visible: type a message in the tab on node-1 and watch it appear in the tabs on node-2 and node-3.
In Part 2 of the map subscriptions series, we introduced a PostgreSQL map broker that lets your application publish real-time map updates inside a database transaction — removing the dual-write problem when you publish via the broker's SQL function from your own transactions. That capability applied only to map subscriptions — keyed state like leaderboards, collaborative boards, and inventories.
\nToday we're extending the same shape to stream subscriptions — the ordered-event primitive that powers notifications, activity feeds, chat messages, audit logs, and order updates. If you have a database row and you want to announce a change in real time, you can now do it atomically with your write — same BEGIN / COMMIT, same outbox architecture, same \"no Redis\" simplicity.
Available in Centrifugo v6.8.0+. The PostgreSQL stream broker is a recent addition — we're eager for feedback. SQL function shapes, configuration keys, and outbox internals may still adjust before they're considered stable.
cf_stream_publish(...) inside the same SQL transaction as your row write — both commit atomically. No outbox table to manage in your app, no CDC pipeline, no dual-write gap.LISTEN/NOTIFY for low-latency wakeup, vacuum-free retention.orders table — with a runnable kitchen-orders demo.getState callback) is covered in App-owned state with stream subscriptions.Integrating a real-time system with a relational database creates the same gap: the backend writes to the database, then publishes to the real-time layer as a separate operation. If the process crashes between them — or if the publish fails — the database and subscribers fall out of sync. Users see stale data until they refresh.
\nWe covered this in depth for map subscriptions. The same problem applies — even more broadly — to stream subscriptions. Every notification system, every audit trail, every order-status feed has the same shape: write a row to your database, then announce the change over WebSocket. The PostgreSQL stream broker lets you combine both into one transaction.
\nCentrifugo creates a cf_stream_publish SQL function when the PostgreSQL stream broker initializes. Your application calls it inside its own transaction:
BEGIN;
-- Business logic: update order status
UPDATE orders SET status = 'shipped', updated_at = NOW()
WHERE id = 42;
-- Publish to real-time channel (same transaction)
SELECT * FROM cf_stream_publish(
p_channel := 'orders:42',
p_data := '{\"status\": \"shipped\"}'::jsonb
);
COMMIT;
If the transaction rolls back, the real-time update never happened. No outbox table to manage in your application, no CDC pipeline, no eventual consistency — just a single transaction. The architecture is the same outbox pattern we use for map subscriptions: all writes land in PostgreSQL tables atomically, and Centrifugo's outbox workers pick up new entries and deliver them to subscribers. When use_notify is enabled, delivery latency drops to low single-digit milliseconds.
The transactional guarantee applies to callers using the SQL function path — i.e. your backend code calling cf_stream_publish directly inside its own SQL transaction alongside the row write. Publishes that go through Centrifugo's HTTP/GRPC API remain a separate operation from your DB write (the historic dual-write shape) — same as before. The SQL function path is what removes that gap; it's an additional integration option, not a change to existing publish APIs.
A kitchen-orders system shows the shape well: a single orders table shared by all restaurants, and one channel per restaurant — kitchen:{restaurant_id} — so each kitchen display only sees its own. The demo below is runnable from the pg_stream_broker example on GitHub:
Every write to a restaurant's orders commits atomically with a publish on that restaurant's channel:
\nBEGIN;
INSERT INTO orders (id, restaurant_id, status, items, updated_at)
VALUES (7001, 42, 'received', $1, NOW());
SELECT cf_stream_publish(
p_channel := 'kitchen:42',
p_data := '{\"order_id\":7001,\"status\":\"received\",...}'::jsonb
);
COMMIT;
Status transitions (received → preparing → ready → served) do the same — UPDATE orders plus cf_stream_publish(p_channel := 'kitchen:42', …) in one transaction. The application follows one rule: every code path that mutates a row for restaurant X emits the publish for kitchen:X in the same transaction.
Each channel has its own meta row and its own top offset; writes on kitchen:99 never block or interfere with kitchen:42. This scales to thousands of tenants on a single PostgreSQL — each channel is an independent append-only stream, and the shared cf_stream partitioned table absorbs the union.
The kitchen demo has a property worth naming: the orders table is the only source of truth. There's no duplicate state table and no broker-managed snapshot — Centrifugo streams the change events, and the stream broker keeps just a thin bridging window in cf_stream (the partition retention window) while the app database owns everything historical.
The client side of that pattern — loading the current orders and staying in sync across reconnects — is handled by the SDK's getState callback for stream subscriptions, and it is a topic of its own. App-owned state with stream subscriptions walks through it, including the read path of this kitchen demo and a second shape where an aggregator fronts a Kafka feed.
On a local PostgreSQL 16 (Homebrew, Apple M4):
\n| Operation | Result |
|---|---|
| Publish | ~17,000 ops/sec |
| Publish → delivery latency | ~2 ms |
| Partition drop (10K rows) | ~1 ms |
These numbers are from a single Centrifugo instance running the broker's Go integration tests in benchmark mode against the same machine's PostgreSQL — small JSON payloads, default broker configuration, parallel goroutines exercising cf_stream_publish. They're rough estimates, not numbers you can take to production: real workloads vary with payload size, connection pool, network latency, and your PostgreSQL's own write capacity. In production, multiple Centrifugo nodes and application instances call cf_stream_publish concurrently — aggregate throughput scales with the number of writers up to PostgreSQL's own write capacity. For notification, audit-log, and order-update workloads this is plenty of headroom. For ultra-high-volume telemetry that doesn't need transactional publishing, the Redis broker remains the right choice.
Configure the PostgreSQL stream broker as your Centrifugo broker:
\n{
\"broker\": {
\"enabled\": true,
\"type\": \"postgres\",
\"postgres\": {
\"dsn\": \"postgres://user:pass@localhost:5432/app?sslmode=disable\",
\"use_notify\": true,
\"partition_retention_days\": 7
}
}
}
The broker automatically creates the required tables and SQL functions on startup. Call cf_stream_publish from your application's SQL transactions to publish atomically.
Read the full stream broker documentation for configuration reference, and see the map subscriptions Part 2 post for the outbox architecture deep-dive that both brokers share.
", "url": "https://centrifugal.dev/blog/2026/05/24/pg-stream-broker-benefits", "title": "Transactional publishing for stream subscriptions with PostgreSQL", "summary": "The PostgreSQL stream broker brings transactional publishing to Centrifugo's stream subscriptions. Real-time updates commit alongside the database write that triggers them — same SQL transaction, no application-side outbox, no CDC pipeline, no separate publish API call.", "date_modified": "2026-05-24T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "postgresql", "streams", "outbox" ] }, { "id": "https://centrifugal.dev/blog/2026/05/23/map-subscriptions-part-2", "content_html": "In the previous blog post we introduced Map Subscriptions. We mentioned that Centrifugo has PostgreSQL Map Broker, in this post we are providing more details about it. The PostgreSQL map broker allows publishing to a Centrifugo Map within an application SQL transaction:
\nBEGIN;
-- Update application data
UPDATE board_items SET data = '{\"text\": \"Updated card\"}'::jsonb
WHERE board_id = 123 AND item_id = 'card_42';
-- Publish to real-time channel (same transaction)
SELECT * FROM cf_map_publish(
p_channel := 'boards:123',
p_key := 'card_42',
p_data := '{\"text\": \"Updated card\"}'::jsonb
);
COMMIT;
If the transaction rolls back, the real-time update never happened. No outbox table to maintain, no CDC pipeline, no eventual consistency — just one PostgreSQL transaction. The rest of this post unpacks how it works.
\nPart 1 introduced map subscriptions and the memory and Redis brokers. This post focuses on the PostgreSQL broker, new in Centrifugo v6.8.0.
\nCentrifugo creates SQL functions (cf_map_publish, cf_map_remove) that your application calls inside its own database transactions. The map state update and any other writes you do alongside commit or rollback together — atomically (as shown in the opening example).
This guarantee applies to callers using the SQL function path — your backend code calling cf_map_publish directly inside its own SQL transaction. Publishes that go through Centrifugo's HTTP/GRPC API are still a separate operation from your DB writes (the historic dual-write shape). The SQL function path is what makes them one transaction; it's an additional integration option, not a change to the existing publish APIs.
The following demo shows a polls feature where each vote is a PostgreSQL transaction that atomically updates the result and publishes to Centrifugo (one of many examples in the map demo collection):
\n\nA common approach to keeping a database and an external system in sync is CDC — Change Data Capture from the PostgreSQL write-ahead log. Supabase Realtime uses this model: it reads committed changes from the WAL and pushes them to clients. This approach requires either external tooling (Debezium, Kafka Connect) or specialized infrastructure that understands the WAL format. It also means CDC sees changes only after they happen — it can't be part of the write itself.
\nWe initially built a WAL-based version, but removed it. The reason: the main bottleneck in this architecture is PostgreSQL's write throughput — how fast your application can commit transactions. Reading committed changes from an outbox table is cheap by comparison, and PostgreSQL's LISTEN/NOTIFY keeps delivery latency low — typically under a few milliseconds. WAL-based CDC solves a read problem we don't actually have here, while adding real complexity — logical replication slots, WAL parsing, schema coupling. It also requires wal_level = logical — a setting not every PostgreSQL deployment has enabled, and one that some managed providers restrict or charge extra for. The outbox pattern keeps everything in regular SQL tables — no external dependencies, no additional infrastructure between PostgreSQL and Centrifugo.
The outbox pattern also gives us something WAL-based CDC can't: the ability to write both the real-time state (cf_map_state) and the change stream (cf_map_stream) atomically within the same SQL function call. With CDC, the system reacts to what was written. With the outbox, Centrifugo's SQL functions control what gets written — including conditional logic like if_new, if_exists, and compare-and-swap — all inside the transaction.
When your transaction calls cf_map_publish, the function does three things in a single atomic operation:
cf_map_state — the current key-value snapshotcf_map_stream — the ordered change log that outbox workers readcf_map_meta — the offset and epoch that track where the stream isOrdering is the tricky part: the outbox worker advances a simple cursor over the stream table's auto-increment id, so it must never see a higher id committed while a lower one is still in flight — that would be a gap it can't safely skip. The function handles this with a per-shard serialization lock: channels are hashed across shards (num_shards, default 8), and every publish first locks its shard's row and holds it until commit. Within a shard, ids are therefore assigned and committed in the same order, while different shards proceed in parallel. Without this lock, two concurrent transactions — even publishing to different channels — could insert rows with ids 5 and 6 but commit in reverse order: the worker would see id 6 appear while id 5 is still uncommitted. The channel's meta row is locked next (lock order: shard → meta → state) to assign the per-channel offset and protect against concurrent meta cleanup. The cost of this design is that writers within a shard briefly queue behind each other — the price of a gap-free cursor — and num_shards tunes that parallelism.
Centrifugo runs a pool of outbox workers — one per shard (num_shards, default 8). Each channel is assigned to a shard by hash, and each worker independently polls its portion of the stream table using an in-memory cursor that tracks the last delivered stream id. Workers are stateless — on restart the cursor starts from the current tail of the stream, and no position is persisted. Delivery guarantees don't depend on the workers at all: every publication carries a per-channel offset and epoch, so a client that missed messages — whether due to its own reconnect or a Centrifugo node restart — recovers them from the stream table on resubscribe, and fresh subscribers load the current state from the snapshot table.
By default, workers poll every 100ms. Enabling use_notify triggers PostgreSQL's LISTEN/NOTIFY when new entries are committed, waking the worker immediately — reducing delivery latency to low single-digit milliseconds. Every Centrifugo node runs its own set of workers, so delivery continues even if a node goes down.
In effect, the broker-owned collection lives durably in PostgreSQL with the same operational story as the rest of your data — backups, monitoring, psql access — and clients see it live over WebSocket. No additional message broker, no new data pipeline.
The stream table is automatically partitioned by day. Old partitions are dropped entirely — instant, no row-by-row deletion, no expensive vacuum operations. This is built into the open-source broker via partition_retention_days (default 7) and partition_lookahead_days (default 2). No manual maintenance needed — the broker pre-creates future partitions and drops old ones on a regular interval.
On PostgreSQL 16 (Apple M4, native install — not Docker):
\n| Operation | Result |
|---|---|
| Map publish | ~16,000 ops/sec |
| Map publish with CAS | ~11,000 ops/sec |
| Idempotent publish | ~17,500 ops/sec |
| Read state (full) | ~10,800 ops/sec |
| Read state (paginated) | ~50,000 ops/sec |
| Remove | ~42,000 ops/sec |
| Publish → delivery latency | ~1.3 ms |
All publish operations go through a single SQL function call (cf_map_publish) that atomically updates the state table, appends to the stream, and increments the channel position. ~16,000 publishes per second per broker is enough for collaborative state workloads — boards, inventories, presence.
These numbers come from the broker's Go integration tests in benchmark mode against a same-machine PostgreSQL — small JSON payloads, default broker configuration, parallel goroutines exercising the SQL functions. They're rough estimates, not numbers you can take to production: real workloads vary with payload size, connection pool sizing, network latency, and your PostgreSQL's own write capacity. For context, the Redis map broker on the same hardware would be faster per-operation, but doesn't offer transactional publishing. The numbers above reflect a single Centrifugo instance with parallel goroutines. In production, multiple Centrifugo nodes and application instances publish concurrently — aggregate throughput scales with the number of writers up to PostgreSQL's own write capacity.
\nTransactional publishing is currently experimental — we may adjust the SQL function API and outbox architecture based on feedback. We've published several PostgreSQL-backed demos in the map demo collection, including a sprint board that demonstrates transactional publishing with Docker Compose.
\nPart 1 covers the full map subscriptions design — sync protocol, modes, and broker overview. And check out the companion post on shared poll subscriptions — the other new subscription type we're introducing alongside map subscriptions.
\nRead the full map subscriptions documentation for configuration reference, PostgreSQL broker setup, and transactional publishing examples.
", "url": "https://centrifugal.dev/blog/2026/05/23/map-subscriptions-part-2", "title": "Map subscriptions (Part 2) — when your PostgreSQL transaction is your real-time publish", "summary": "The PostgreSQL map broker brings durable persistence to Centrifugo-owned key-value collections and lets your application publish updates atomically inside its own database transactions — no dual-write, no application-side outbox to maintain.", "date_modified": "2026-05-23T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "websocket", "state-sync", "postgresql" ] }, { "id": "https://centrifugal.dev/blog/2026/05/22/map-subscriptions", "content_html": "A map subscription — new in Centrifugo v6.8.0 — is a real-time key-value collection that Centrifugo manages. The map broker stores the entries; the SDK keeps a live/realtime mirror on every subscribed client.
\nSubscribe – and you get the current snapshot, then live updates as keys updated/removed, then automatic re-sync on reconnect. No separate REST endpoint to load initial state, no race window between HTTP and WebSocket. Convergence is part of the protocol — no recovered: false flag to handle in app code.
Centrifugo's original model is stream subscriptions — channel-based pub/sub with optional history and recovery so reconnecting clients catch up on missed publications. They work well for chat, notifications, activity feeds, and anywhere clients need an ordered sequence of events, and they remain the right choice for most real-time features.
\nFor the broad case of \"I have data in my own database and want clients to see it live\", a stream subscription with a getState callback reading from your own tables is usually the natural fit — your schema stays the source of truth. Map subscriptions fit a different shape: collections without an obvious home in the application's database — cursors that exist for a few seconds at a time, presence sets, IoT device state, lobby members, feature flags, live dashboards. Each is conceptually just a key-value collection that should be live in the browser. Building a store, a change feed, and a snapshot endpoint for each one is a lot of infrastructure for that. Map subscriptions are that primitive, baked into Centrifugo.
Centrifugo also offers a new presence implementation built on map subscriptions, with all the same benefits — automatic synchronization and a clean SDK API.
\nConsider two examples. First, shared cursors: each user publishes their cursor position, and every other user sees it move in real time. You can publish coordinates through a stream subscription, and that works — but after a page reload, the new client has no way to get the current position of all cursors. It must wait for each user to move again. Presence gives you the list of who's in the channel, but not their associated state.
\n\nSecond, a live scoreboard. The client doesn't care about the history of score changes — it needs the current state of all matches, and it needs updates as they happen. You can absolutely build this with stream subscriptions — we did exactly that in our real-time leaderboard tutorial using Redis, cache recovery, and delta compression. It works well. The challenge is the initial load: there's a gap between the REST response and the moment the subscription starts, and updates published during that gap can be missed.
\nWe've explored this problem before and even provided a RealTimeDocument helper class that manages versioning, re-fetches state on gaps, and reconciles late-arriving updates. It works, but we kept seeing teams building collaborative features, live dashboards, or presence systems implementing similar logic on top. We wanted a more convenient SDK API for this pattern — one where state delivery and recovery are built in, so application code doesn't have to manage them.
For use cases like scoreboards, presence, or collaborative state, what matters is the current collection of entries — not the ordered sequence of events that stream subscriptions provide. Map subscriptions are a new subscription type for this: instead of an append-only publication stream, a map channel maintains a synchronized collection — a set of key-value entries where each entry can be independently published, updated, or removed. When a client subscribes, it receives the full state through a paginated protocol, then transitions to live updates. The SDK handles all of this transparently — the application just reacts to sync (full state ready) and update (single entry changed) events.
const sub = client.newMapSubscription('dashboard:main');
sub.on('sync', (ctx) => {
renderDashboard(ctx.entries);
});
sub.on('update', (ctx) => {
ctx.removed ? removeEntry(ctx.key) : upsertEntry(ctx.key, ctx.data);
});
sub.subscribe();
The subscription itself delivers the state — there's no separate REST call, no gap to bridge. Compare this with stream subscriptions, where the application receives individual publication events and must build state from the event sequence. With map subscriptions, the SDK maintains the collection internally — the application just renders what it's given. After any disconnect, clients always converge to the correct state: the SDK either catches up from the stream or re-syncs from scratch, then emits a fresh sync event. There's no recovered: false flag to handle — convergence is guaranteed by the protocol.
Map subscriptions are configured through the same channel namespace system as stream subscriptions — you set subscription_type: \"map\" on a namespace, configure the mode and options, and channels in that namespace become map channels. The same connection, the same client SDK, the same authentication and authorization flow. Existing stream namespaces are unaffected. The client SDK provides a dedicated MapSubscription type with narrowed methods and events — sync, update, mapPublish, mapRemove — rather than overloading the existing Subscription with mode flags. The same approach applies to presence variants (MapClientsSubscription, MapUsersSubscription).
Underneath the simple-looking API there's a real protocol challenge: the client must paginate through potentially large state while new updates keep arriving. This is the same race condition that RealTimeDocument addressed — but now solved at the protocol level, so application developers don't have to think about it.
The subscription goes through three phases:
\nDuring the transition from stream to live, the server buffers incoming publications, merges them with recovered stream entries, and checks for continuity — reusing the same buffering and merge infrastructure that powers stream recovery in regular channels. If continuity is broken, the client re-syncs from scratch automatically. The SDK handles all three phases internally — the application never sees pagination cursors or stream offsets.
\nNot all state has the same lifecycle. Cursor positions should disappear seconds after a user disconnects. Session data should survive brief network interruptions but not persist forever. Inventory entries should persist until explicitly removed.
\nStream subscriptions offer fine-grained control over these concerns — history_size, history_ttl, force_recovery can be configured independently. Map subscriptions take a different approach, bundling them into three modes that cover the most common state sync patterns:
Ephemeral — no stream history, entries expire via TTL. On reconnect, the client gets a full state snapshot. This is the most lightweight option — about 35-40% less work per publish compared to modes with a stream. Best for cursors, typing indicators.
\nRecoverable — stream history with TTL-expiring entries. On reconnect, the client catches up from the stream instead of re-fetching everything. Best for presence, sessions, polls, game lobbies — data that auto-expires but needs efficient recovery.
\nPersistent — stream history with permanent entries. Data lives until explicitly removed. Best for inventories, collaborative documents, dashboards — permanent state with efficient reconnect.
\nThe mode determines which phase of the sync protocol is used on reconnect: ephemeral always re-syncs from state, recoverable and persistent attempt stream catch-up first.
\n\nHere's the protocol visualizer demo — it shows all three phases in action, including paginated state delivery, stream catch-up, and the transition to live updates:
\n\nRecoverable and persistent modes support Fossil delta compression — deltas are computed per key, between successive values of the same entry, so clients receive compact patches instead of full payloads when only part of the data changes.
\n\nSome patterns need writes that can fail gracefully — two players claiming the last slot in a game lobby, two bidders placing an auction bid at the same instant. Map subscriptions support two forms of conditional writes. Key modes — if_new (only insert if key doesn't exist) and if_exists (only update if key exists) — cover slot claiming and heartbeat-only updates. For stronger guarantees, compare-and-swap via ExpectedPosition checks the channel's stream offset and epoch before writing — if another write happened since the client's last read, the operation is rejected and the client can retry with fresh data.
The same keyed structure applies directly to presence. Centrifugo has had presence since the early days — you can query who's in a channel and receive join/leave events. This works well for channels with moderate participant counts. For channels with thousands of participants, though, returning the entire list in a single response becomes expensive. And join/leave events are delivered with at-most-once guarantee — fine for live indicators, but there's no built-in way to catch up on events missed during a disconnect.
\nMap subscriptions work well for these larger-scale scenarios. Centrifugo provides two presence subscription types:
\nmap_clients — one entry per connection (key = client ID). When a client unsubscribes or disconnects, its entry is removed immediately.map_users — one entry per user (key = user ID). A user may have multiple connections, so entries can't be removed on a single disconnect — they expire via TTL after the last connection for that user leaves the channel.Because these are regular map channels, clients get paginated state on subscribe and live join/leave updates in real time. Recovery works the same way — reconnecting clients catch up from the stream instead of re-fetching everything.
\nThis is configured through channel prefixes. When a client subscribes to game:abc, the server can automatically publish presence entries to clients:game:abc (per-connection) and users:game:abc (per-user). These are separate map channels that other clients can subscribe to independently:
Client subscribes to game:abc
│
├──► auto-publish to clients:game:abc (key = client_id)
└──► auto-publish to users:game:abc (key = user_id)
Other clients subscribe to clients:* / users:* to see who's online
{
\"channel\": {
\"namespaces\": [
{
\"name\": \"game\",
\"map_clients_presence_channel_prefix\": \"clients:\",
\"map_users_presence_channel_prefix\": \"users:\"
},
{
\"name\": \"clients\",
\"subscription_type\": \"map_clients\",
\"map\": { \"mode\": \"recoverable\", \"key_ttl\": \"60s\" }
},
{
\"name\": \"users\",
\"subscription_type\": \"map_users\",
\"map\": { \"mode\": \"recoverable\", \"key_ttl\": \"60s\" }
}
]
}
}
Here's a game lobby demo that shows presence built on top of map subscriptions — the sidebar tracks connected players in real time using MapClientsSubscription:
A handful of players is the easy case. The same primitive scales much further — here are two browser tabs subscribed to the same map_clients channel while the population is being filled to 100,000 connected members. Each cell in the grid is one connection; green flashes mark joins, red flashes mark leaves. The tabs connect during the fill, so they have to reconcile a moving target — paginated initial state plus the joins still happening in real time. Once the population stabilizes both tabs show the same converged 100k. The clip ends with a reconnect: state paginates back at 10k entries per page, the full grid repaints quickly, and the tab is live again:
To be clear: a hundred thousand members in a single presence channel is not a typical production shape and we don't expect most apps to need it. This is a deliberately extreme demo showing what the protocol is capable of — not a recommended pattern. Most real systems work with channels of dozens to a few thousand participants, where the main advantages of map presence are convergence on reconnect and join/leave reliability, rather than raw count. (For the curious: we ran the same setup at 1,000,000 members and a single browser tab still synchronizes — initial state takes around twenty seconds to paginate, then live updates flow normally. Past that point the bottleneck shifts from the protocol to the cost of streaming the snapshot itself.) Both 100k and 1M variants are runnable from the v6/map_presence_demo example.
The recoverable mode is what makes map-based presence more capable than Centrifugo's traditional presence. With recoverable mode, reconnecting clients catch up from the stream rather than re-fetching the full participant list. With ephemeral mode, clients would get a full snapshot on every reconnect — which is the same behavior traditional presence already provides, losing the convergence advantage.
Map subscriptions need a backend to store state and coordinate updates. Centrifugo supports three map brokers, each suited to different deployment scenarios.
\nMemory is the default — zero dependencies, single-node, state lost on restart. Good for development and ephemeral data on single-node deployments.
\nRedis adds distribution across nodes. We've invested years into making Centrifugo's Redis engine efficient — connection pooling, pipelining, client-side consistent sharding, atomic Lua scripts. The Redis map broker builds on that same foundation, reusing the existing connection infrastructure and following the same patterns. State is stored in Redis hashes, with atomic Lua scripts ensuring that state update, stream append, and PUB/SUB broadcast happen as a single operation — the same atomicity approach we use for stream subscription history. This is the typical choice for ephemeral and recoverable modes in multi-node setups.
\nPostgreSQL goes further. In a recent survey, 86% of Centrifugo users reported having PostgreSQL in their production stack, with 76% using it as their primary database. The PostgreSQL broker stores map state in regular SQL tables and enables transactional publishing — real-time updates that commit or roll back inside your database transactions, eliminating the dual-write problem entirely.
\nHere's a sprint board demo where cards are moved between columns — each drag-and-drop is a PostgreSQL transaction that updates the board state and publishes to Centrifugo atomically:
\n\nMost real-time messaging systems are built around PUB/SUB — Pusher, Socket.IO, NATS all center on message delivery. A growing number do offer built-in state synchronization. Firebase Realtime Database and Supabase Realtime sync data to clients directly from their managed databases. Ably has expanded beyond pub/sub with LiveSync and LiveObjects for database-to-client sync and collaborative state primitives. Liveblocks provides collaborative state with CRDTs. But these are either cloud-only services tied to a specific database — where you can only integrate as deeply as the platform allows — or specialized tools for a narrow use case like collaborative editing.
\nWhy not CRDTs? Because we want to stay consistent with what Centrifugo has always been: a generic real-time transport that doesn't interpret the data it carries. Centrifugo delivers JSON or binary payloads without interpreting their contents — it doesn't parse your data, doesn't merge it, doesn't resolve conflicts at the field level. The entire protocol — including map subscriptions — works over both JSON and Protobuf, so latency-sensitive or bandwidth-constrained applications can use compact binary encoding end to end. CRDTs require the transport layer to understand the data structure and apply merge semantics, which ties the system to specific data types. Map subscriptions work the same way as stream subscriptions: Centrifugo synchronizes opaque key-value entries — your application decides what they contain and how to interpret them. This keeps the system generic and the design consistent across all three subscription types.
\nCentrifugo occupies a different spot because it's self-hosted — it runs in your infrastructure, connects to your databases, and calls your backend directly. That proximity opens up features a cloud service sitting between you and your users can't really offer — from transactional publishing (where the publish is part of your DB transaction, not an after-the-fact CDC reaction) to the low-latency proxy system that powers subscribe authorization and shared poll refresh.
\nFor scenarios that need per-item access control within a single map channel, Centrifugo PRO adds a server-side publication tags filter — your backend assigns tags to entries and sets a filter per subscriber via the subscribe proxy or JWT. Only matching entries are delivered, across all sync phases. This enables RBAC patterns without splitting data into separate channels per access scope.
\nStream subscriptions, map subscriptions, and shared poll subscriptions cover three different ways clients relate to data — ordered events, synchronized collections, and polled read-only state — while staying payload-agnostic across the board. Your application decides what the data means; Centrifugo handles delivery. Switching between primitives is a namespace configuration choice, not an architectural one.
\nMap subscriptions are currently experimental — we may adjust the API, configuration, and protocol based on feedback. At this point, only centrifuge-js supports map subscriptions on the client side. We plan to extend support to other SDKs.
We designed map subscriptions to share everything stream subscriptions already have — not a separate system grafted on. Same namespace configuration, same Redis infrastructure, same client SDK connection, proxy system, recovery internals, and transport layer. The goal is that adopting map subscriptions should feel like switching a namespace option, not adopting a different system.
\nAnd check out the companion post on shared poll subscriptions — the other new subscription type we're introducing alongside map subscriptions.
\nWe've published a collection of interactive demos covering different map subscription features — from ephemeral cursors to PostgreSQL-backed sprint boards. Each demo runs with Docker Compose and showcases a different aspect of the feature.
\nRead the full map subscriptions documentation for configuration reference, broker setup, and client SDK API details.
", "url": "https://centrifugal.dev/blog/2026/05/22/map-subscriptions", "title": "Map subscriptions (Part 1) — synchronized key-value state and presence", "summary": "A new subscription type in Centrifugo — real-time key-value collections with paginated state delivery, guaranteed convergence on reconnect, per-key TTL, conditional writes, and Fossil delta compression.", "date_modified": "2026-05-22T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "websocket", "state-sync", "presence" ] }, { "id": "https://centrifugal.dev/blog/2026/05/21/two-million-particles-on-centrifugo", "content_html": "David Gerrells wrote a blog post How fast is Go - simulating millions of particles on a smart TV — describing a Go server that simulates two million particles in a 2200 × 2200 world at 60 Hz, ships frames to clients at 30 Hz over WebSocket, and lets anyone connected pull particles around with their cursor. The transport is hand-written for speed: bit-packed binary frames, manual protocol, raw WebSocket library. The live demo runs at howfastisgo.dev — try it before reading on.
\n\n\nDavid's goal was to explore Go performance. Once we saw the demo we immediately wanted to try reproducing it on top of Centrifugo — to see whether a generic real-time transport like Centrifugo could carry this kind of payload. At first glance it looked straightforward: Centrifugo provides a binary WebSocket transport, and the simulation already runs on the server. But along the way we ran into design differences that meant we couldn't quite match the original's per-viewer bytes on the wire. We'll show what we built, what it cost, and why the overhead is worth it for UX and scalability.
\nSource code of our final demo: v6/millions_of_particles.
The \"two million particles\" lives entirely on the server. What goes to clients is a density map — one bit per world cell, answering \"is there any particle in this cell?\". Several particles in the same cell collapse to one bit. Bytes per frame scale with viewport pixels, not particle count — bumping the simulation to 4M particles wouldn't change the wire size at all, the cells would just get fuller.
\nThe server runs everything in one Go process: the simulation, the WebSocket connections, and per-client camera state. On every tick it walks the connected clients, reads each one's camera (x, y, width, height) shipped up from the browser, crops that rectangle from the world buffer, and writes the bit-packed bytes straight to that client's WebSocket. A typical desktop window of 1410 × 730 pixels packs to 1410 × 730 / 8 ≈ 129 KB per tick — that's all a viewer ever receives. The client sees about 21% of the world and pans by changing the camera; cursor input flows back up the same WebSocket and pulls particles around in the simulation.
That tight coupling — sim, sockets, and per-client cameras all in the same process — is what keeps the original lean per viewer: the server cuts a custom message for each connection because everything it needs is right there. Now let's see what changes when we put a generic broker in the middle.
\n ┌────────────────────────────────┐
│ Go backend (particle sim) │
│ 60 Hz sim · 30 Hz publish │
└────┬──────────────────────▲────┘
│ │
│ POST /api/publish │ HTTP RPC proxy
│ (frame bytes) │ (cursor input)
▼ │
┌────────────────────────────────┐
│ Centrifugo │
│ single channel · WS fan-out │
└────┬──────────────────────▲────┘
│ binary WS frames │ WS RPC (cursor)
▼ │
┌───┐ ┌───┐ ┌───┐ ┌───┐ │
│ B │ │ B │ │ B │ │ B │─┘ browsers
└───┘ └───┘ └───┘ └───┘
The backend publishes one binary payload per tick to a single Centrifugo channel; every subscribed browser receives the same WebSocket frame and slides a local camera over the bitmap to render its slice. Cursor input flows back via RPC over WebSocket, proxied to the backend's HTTP endpoint.
\nThe publisher's job is constant: pack the whole 2200 × 2200 world at 1 bit per pixel, send to a channel — and that's it – Centrifugo handles the fan-out. Pan works locally too: each browser has the whole world in memory and just slides a camera over the bitmap.
\n\nSee in better quality on YouTube.
\nIt works — but at roughly 5× the bytes the original sends to each viewer (~605 KB vs. ~129 KB). It's by design: Centrifugo is a standalone broker. It doesn't know about user cameras, viewports, or which slice each viewer cares about, so a single channel has to ship bytes useful for any subscriber, and the simplest \"useful\" is the whole world.
\nAnd the gap widens with world size. Bump the world to 10000 × 10000 and the naive port ships ~12.5 MB per viewer per tick, while the original would still send ~129 KB — each viewer only pays for their viewport. The naive approach scales with world size; the original scales with viewport size. So we have a real reason to find a better fit.
\nWe could try one channel per viewer, with the backend tracking each camera and packing an individual crop per tick. That would get us closer to the original's ~129 KB per viewer, but it gives up fan-out — the thing Centrifugo is built for — and turns every viewport change into RPC traffic up plus a per-client publish down. The backend would also need to track camera state per connection — easy in the original's single-process design, awkward when Centrifugo sits in between.
\nSo we want fewer bytes per viewer and keep fan-out.
\nThe idea: split the world into tiles, and let each viewer subscribe only to the tiles its viewport is touching. The publisher still packs the world once per tick; Centrifugo only delivers each tile to viewers tracking it.
\nA tile is a fixed-rectangle chunk of the world. The grid is set up front — same tiles for every viewer. Every tick the publisher packs each tile in the same bit-packed format used for the whole world before (just smaller, and many of them). Viewers subscribe to the keys for the tiles their viewport currently overlaps; Centrifugo only delivers each tile to viewers tracking it. As the camera moves, the tracked set shifts to follow.
\nThree things to settle in this design: pan behavior at tile boundaries, what tile size to pick, and how to actually deliver tiles to viewers. In that order.
\nPan. A viewer's camera can move at any moment, and when the viewport crosses a tile boundary the newly-revealed tile wouldn't be subscribed yet — we'd see a strip of empty pixels until the next publish landed. Visible pop-in on the leading edge of the motion.
\nThe fix is a small prefetch margin: pre-track the next ring of tiles around the viewport, so freshly-revealed tiles already have data when they slide into view. Panning stays smooth at the cost of a few extra tiles always being subscribed.
\n\nTile size. The savings come from the tiles a viewer doesn't receive — so the tile count needs to be high to make a real dent. A 4 × 4 grid barely helps; a 32 × 32 grid (1024 tiles) tracks the viewport tightly enough to land within striking distance of the original's per-viewer bytes.
\nDelivery. The natural first idea is one channel per tile — tile:0:0, ..., tile:31:31. Each viewer subscribes to the tile channels its viewport currently covers; the publisher publishes to 1024 channels per tick. This works. The cost shows up every time the viewer pans: crossing a tile boundary means sending a subscribe message for each newly-entered tile and an unsubscribe for each one left behind. Each subscribe also goes through its own authorization check, because the channels are independent — there's no way to share auth across them, even though for our purposes all 1024 tiles are really parts of one thing. So even small viewport movements turn into a lot of subscribe/unsubscribe traffic.
But in Centrifugo v6.8.0 we have a mechanism that seems to fit better here — Shared Poll subscriptions.
\nA subscriber on a Shared Poll channel calls track(keys) to declare which keys it cares about. A key is just an identifier tied to some application entity — in our case, a single tile.
What makes Shared Poll fit here is its tracking model, not the polling itself: one channel, many keys, with each viewer declaring which keys it cares about. The name comes from its main delivery mode — Centrifugo aggregates tracked keys and asks the backend for updates on a timer — but it also has a direct publish path (shared_poll_publish), which is what we will use here to have low latency world updates. Centrifugo only delivers a key's bytes to viewers currently tracking that key. That's the property we need.
On pan, the client doesn't subscribe to anything new — it just updates its track set with untrack(leaving) and track(entering). Centrifugo handles the rest.
For our particle demo it maps cleanly:
\ntiles:world with shared-poll subscription type, 1024 keys (t_<tx>_<ty>)./api/batch request — 1024 shared_poll_publish commands, one HTTP round-trip instead of 1024.The publisher side. A single monotonic version counter for all tiles, and a per-process epoch generated on startup so that a backend restart triggers Centrifugo to invalidate connected subscribers (they auto-resubscribe and pick up fresh state without a page reload):
\nchannelEpoch := uuid.NewString()
var globalVersion uint64
// On every tick:
v := atomic.AddUint64(&globalVersion, 1)
items := make([]SharedPollItem, 0, len(tilePayloads))
for i, payload := range tilePayloads {
items = append(items, SharedPollItem{
Key: TileKey(i % TilesPerSide, i / TilesPerSide),
Data: payload,
Version: v,
})
}
api.BatchSharedPollPublish(ctx, \"tiles:world\", channelEpoch, items)
The batch wrapper turns that into one POST /api/batch carrying 1024 commands:
func (c *CentrifugoAPI) BatchSharedPollPublish(ctx context.Context, channel, epoch string, items []SharedPollItem) error {
commands := make([]map[string]any, 0, len(items))
for _, it := range items {
commands = append(commands, map[string]any{
\"shared_poll_publish\": map[string]any{
\"channel\": channel,
\"key\": it.Key,
\"b64data\": base64.StdEncoding.EncodeToString(it.Data),
\"version\": it.Version,
\"epoch\": epoch,
},
})
}
return c.call(ctx, \"batch\", map[string]any{\"commands\": commands})
}
For per-key authorization the SDK calls a getSignature callback whenever the tracked key set changes. The callback hits a backend endpoint that returns an HMAC over (channel, keys, expiry) — the same shape the drones demo uses:
func MakeTrackSignature(secret, channel string, keys []string, user string, ttlSec int) string {
now := time.Now().Unix()
expiry := now + int64(ttlSec)
keysHash := sha256.Sum256([]byte(strings.Join(keys, \"\\x00\")))
payload := fmt.Sprintf(\"%d:%d:%s:%s:%x\", now, expiry, user, channel, keysHash)
mac := hmac.New(sha256.New, []byte(secret))
mac.Write([]byte(payload))
return fmt.Sprintf(\"%d:%d:%x\", now, expiry, mac.Sum(nil))
}
The viewer ties it together. One subscription, a track set that follows the camera, and pan reduces to changing the set:
\nconst sub = centrifuge.newSharedPollSubscription('tiles:world', {
getSignature: async (ctx) => {
const resp = await fetch('/api/track_refresh', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ keys: ctx.keys, channel: 'tiles:world' }),
});
return resp.json();
},
});
sub.on('update', (ctx) => {
const m = /^t_(\\d+)_(\\d+)$/.exec(ctx.key);
const tx = +m[1], ty = +m[2];
applyTile(tx, ty, ctx.data); // decode this tile's bytes into the world buffer
});
function updateTracking() {
const want = visibleTileKeys(); // viewport ∩ tile grid + 1-tile margin
const toTrack = [...want].filter(k => !trackedKeys.has(k));
const toUntrack = [...trackedKeys].filter(k => !want.has(k));
if (toUntrack.length) sub.untrack(toUntrack);
if (toTrack.length) sub.track(toTrack.map(k => ({ key: k, version: 0 })));
}
applyTile decodes a single tile's packed bitmap into the client's world buffer at the tile's world position. The buffer is allocated for the full 2200 × 2200 world (Uint32Array(2200 * 2200)), but only the regions covered by tracked tiles ever get filled — everywhere else stays zero. A requestAnimationFrame loop copies the camera-cropped slice from this buffer onto a window-sized display canvas every paint. Pan handlers (right-mouse drag, arrow keys, WASD) update the camera, call updateTracking(), and the broker adjusts which tiles flow to this viewer.
The result where we also show movement over the world:
\n\nSee in better quality on YouTube.
\nSame simulation, same MacBook 1410 × 730 viewport. Now let's compare:
\n| Approach | Data Size Per tick | Pan & delivery |
|---|---|---|
| Original (per-client crop, raw WS) | ~129 KB | viewport-only; pan tied to 30 Hz publish rate |
Single channel fanout (whole world) | ~605 KB | whole world cached client-side; pan local at any frame rate |
shared_poll tiles | ~186 KB | viewport tiles + prefetch margin; pan local at browser frame rate |
Each tile is its own WebSocket message (Centrifugo can batch them into one frame); the ~186 KB is the average sum across the tile messages a single viewer receives per 30 Hz tick.
\nThe shared-poll variant lands at 1.44× the original's per-tick bytes on a standard MacBook screen, at full resolution, with full panning support, and with fan-out preserved. Worth unpacking why.
\nWhere the extra bytes come from. Three things stack up:
\n| Component | Bytes/tick | Ratio |
|---|---|---|
| Pure viewport (1410 × 730 / 8) | ~129 KB | 1.00× |
| + Tile alignment (21 × 11 tiles, no prefetch) | ~143 KB | 1.11× |
| + 1-tile prefetch margin (23 × 13 tiles tracked) | ~186 KB | 1.44× |
Each tile is 69 px (⌈2200/32⌉). A 1410 × 730 viewport touches ⌈1410/69⌉ × ⌈730/69⌉ = 21 × 11 = 231 tiles; with a one-tile prefetch ring, 23 × 13 = 299 tiles. At 621 bytes per tile: 231 × 621 ≈ 143 KB without prefetch, 299 × 621 ≈ 186 KB with.
\nTile alignment is the small overhead: your viewport rarely lines up with tile edges, so the tiles around its border load in full even when only a strip of each is actually visible (that's the 1.11× row). Prefetch is the bigger cost, and it's a UX choice: without it, panning into a fresh tile would render zeros until track() lands and the next publish arrives — visible pop-in on the leading edge.
Why smaller tiles don't help. Two effects pull in opposite directions as tile size shrinks:
\nshared_poll_publish frame with key, version, epoch, and framing bytes — plus byte-alignment loss inside the 1bpp packing (a 35-px row still needs 5 bytes, with most of the last byte wasted). Halving tile size yields 4× more tiles, so those fixed costs scale up 4×.The two roughly cancel — the ratio stays near the same value across a wide range of grid sizes. Going larger makes prefetch waste dominate; going smaller makes per-tile overhead dominate. That ceiling is built into the tile model whenever you also want panning to stay smooth — it's not something you can tune away.
\nWhy prefetch exists at all — and why pans feel smoother because of it. The two designs differ in where the world lives on the client.
\nThe original has no client-side world cache. Every frame is the server-packed bitmap of exactly the current viewport. Pan is a server roundtrip — new camera up, new viewport down — so pan latency is 1 RTT + 1 tick, and pan smoothness is bounded by the 30 Hz publish rate. Quick pans visibly stutter.
\nTile mode puts the world partially on the client. Each viewer keeps a local buffer of its tracked tiles and re-crops it on every requestAnimationFrame (60+ Hz, well above the publish rate). Pans inside already-loaded tiles paint instantly from this buffer — that's why pan feels smooth, not anything happening on the wire.
The catch: the cache has edges. Pan into a tile you weren't tracking and you'd render zeros until track() lands and the next publish delivers the bytes — ~33–66 ms of pop-in. Prefetch fills those edges by keeping the next ring of tiles always subscribed.
So prefetch is the price of having a local cache, and the local cache is what makes pans feel smoother. They come together.
\nWhat the broker model gives you. That overhead isn't what you pay for the broker — it's the cost of prefetch + alignment within the broker model. What the broker actually brings doesn't show up in bytes per viewer at all:
\nThe trade: 1.44× bytes per viewer, in exchange for fan-out that doesn't grow with viewer count and multi-node by configuration. The original was built for a different goal — exploring Go performance in a single-process design — and it's excellent at that. A project that needed both 1.0× per-viewer bytes and multi-node would have to build the broker layer itself.
\nBoth modes are wired into a single docker-compose; pick one with the MODE env var:
docker compose up --build # default: fanout (whole world)
MODE=shared_poll docker compose up --build # shared-poll tiles, full-res, panable
We re-implemented the demo in Centrifugo. We didn't match the original's WebSocket bytes — but the trade buys better publish scalability and smoother panning.
\nThe breakdown above pins almost all of the per-viewer overhead on prefetch — drop it and you're at 1.11×, with visible pop-in on the leading edge during fast pans. So the byte overhead is essentially the price of a UX choice we made, not the price of using Centrifugo.
\nShared Poll subscriptions are a clean way to track many objects with a single subscription — track/untrack is one protocol frame for multiple keys, no per-object subscribe handshake. The name comes from the polling delivery mode, but versioned mode also exposes a fast push path (shared_poll_publish), which is what gives us low-latency world updates here.
The full source, both modes selectable via env var, is in v6/millions_of_particles. Again, see David Gerrells' original post and repo.
What if 10,000 clients could stay up to date with just one backend request per second? That's the idea behind shared poll subscriptions, new in Centrifugo v6.8.0.
\nCentrifugo's core model is push-based: your backend publishes to a channel, Centrifugo delivers it over persistent WebSocket connections, and history and recovery catch clients up after a reconnect. This is what we call stream subscriptions — the original and still the most common subscription type. They work well when your backend owns the writes, and they'll remain the foundation of most Centrifugo deployments. But for some use cases a different shape fits better — and gives you things push alone can't.
\n\nO(unique items), not O(clients).SharedPollSubscription and call track() / untrack() as items come into view. Per-key access is enforced via fast HMAC signatures verified locally on every track request.shared_poll_publish for instant push from the write path, and the PRO notification fast path (instant trigger, polled fetch).Consider a social feed where each post displays a live vote count. The data already lives in your database — the question is how to keep it fresh on every client's screen.
\nYou could use stream subscriptions: publish to Centrifugo on every vote and subscribers see the update in real time. This works well for one feature, but it means every write path touching vote counts — every API endpoint, every background job, every migration script — has to call the publish API. As the number of live-updating fields grows (view counts, comment counts, stock levels, configuration flags), the integration surface grows with it. And it assumes you control the writes — third-party APIs, legacy systems, and databases shared across services often don't expose a hook.
\nIn these cases teams reach for polling. Each client hits an endpoint every few seconds — simple, decoupled from writes, works with any data source. But at scale 10,000 clients per second is 10,000 requests per second, mostly returning unchanged data. That overhead is what usually pushes teams toward push-based systems in the first place.
\nShared poll subscriptions move polling from clients to Centrifugo. Clients hold a persistent WebSocket connection (or HTTP-streaming/SSE fallback) and tell Centrifugo which items they care about. Centrifugo asks your backend on a configurable schedule, detects what changed, and pushes updates to interested clients.
\nEach client sees a different subset of items — different pages, scroll positions, search results — but many subsets overlap. Centrifugo aggregates interest across all connected clients on a node and polls only the union of tracked keys once per cycle. If 10,000 clients are watching overlapping sets that cover 200 unique posts, that's one request for 200 items, fanned out — O(unique items), not O(clients).
\n\nYour backend just needs one endpoint that answers \"here is the current data for these keys.\" It's a standard Centrifugo proxy — same shape as subscribe or publish proxies. No publish calls, no event hooks, no coupling to the write path: if you can read the data, you can serve it through shared poll.
\nBefore shared poll, the natural Centrifugo approach would be a channel per item — one for each post's vote count. Channels are lightweight and ephemeral, so creating many is fine server-side. The cost lands on the client: a feed with 50–100 visible posts means 50–100 active subscriptions, each with its own subscribe/unsubscribe frame, signature, recovery state, and position tracking. Across thousands of concurrent users that adds up on both ends of the wire.
\nA single channel for \"all vote updates\" solves the overhead problem but creates a different one: every client receives every update, regardless of which posts they're actually viewing. With millions of posts and thousands of clients, most of the data delivered is irrelevant.
\nShared poll subscriptions give per-item granularity without per-item channels. The client creates one subscription, then tracks and untracks specific keys as the user scrolls:
\nconst sub = client.newSharedPollSubscription('post_votes:feed1', {
getSignature: async (ctx) => {
const resp = await fetch('/api/sign-poll', {
method: 'POST',
body: JSON.stringify({ keys: ctx.keys }),
});
return resp.json();
},
});
sub.on('update', (ctx) => {
ctx.removed ? hideVoteWidget(ctx.key) : updateVoteWidget(ctx.key, ctx.data);
});
sub.subscribe();
sub.track(getVisiblePostIds());
One subscription, arbitrary number of tracked keys; the server only polls items that at least one client is watching. The SDK exposes a dedicated SharedPollSubscription type with track, untrack, and trackedKeys — purpose-built rather than flags on the regular subscription. Server-side, shared poll is configured through the standard channel namespace system: set subscription_type: \"shared_poll\" on a namespace and configure the polling interval, batch size, and backend proxy. Per-key routing is handled internally — hidden behind the same track() call.
Per-item tracking raises an authorization question: how do you control which items a client can track? Centrifugo's existing connection and channel tokens operate at channel level — but a shared poll client subscribes to one channel and tracks many keys within it.
\nShared poll uses HMAC signatures rather than JWTs. Tracked keys are first-class values in the protocol (not buried in a token payload), and HMAC is an order of magnitude faster to generate and verify — which matters when thousands of clients refresh signatures every poll cycle. When the client calls track(), the SDK invokes a getSignature callback; your backend decides which of the requested keys the user is allowed to see and signs that subset. Centrifugo verifies the HMAC on every track request.
The signature binds user, channel, key set, and a time window — it can't be reused for different keys or different users, and it expires. If getSignature returns fewer keys than the client requested, the omitted keys are treated as revoked: the SDK emits removal events and stops tracking them, so revoking access takes effect on the next refresh without any server push. Apply your existing RBAC or row-level security inside getSignature — Centrifugo enforces whatever decisions your backend makes.
The backend refresh endpoint can be as simple or as sophisticated as your data allows. The simplest integration is versionless mode: your endpoint receives a list of keys and returns {key, data} pairs. Centrifugo detects changes by comparing content hashes internally. The backend doesn't need to track versions — it just returns current state.
{
\"result\": {
\"items\": [
{\"key\": \"post_123\", \"data\": {\"votes\": 42}},
{\"key\": \"post_456\", \"data\": {\"votes\": 7}}
]
}
}
This is enough for most use cases. But if your data already has versions (a database sequence number, a timestamp, a monotonic counter), versioned mode lets you do more. Centrifugo includes the last known version in each request, allowing your backend to skip unchanged items — reducing response size and database load. Versioned mode also pairs cleanly with direct publish for instant delivery, since versions coordinate which keys are already fresh.
\nPolling has a latency cost: updates arrive within the polling interval, not instantly. For vote counts and view counts, seconds of latency are fine. But sometimes your backend already has the data right after a database write and wants instant delivery.
\nYour backend can use the shared_poll_publish server API to deliver data directly to tracking clients, skipping the poll cycle entirely. Centrifugo delivers the data immediately and marks the key as \"fresh\" so the next poll cycle skips it, avoiding a redundant backend call.
This gives shared poll two delivery speeds: timer-based polling (seconds of latency, zero integration effort) and direct publish (instant, requires a publish call on the write path). The two complement each other — direct publish for speed, polling as the safety net. If a direct publish is missed (process crash, network issue), the next poll cycle picks it up.
\n\nCentrifugo PRO adds a third option: a notification fast path. Your backend sends a lightweight notification — just the channel and the keys that changed, no data — and Centrifugo immediately polls those specific keys from the backend, outside the regular cycle. This is useful when your backend knows which keys changed (e.g., from a database trigger or webhook) but doesn't have the data ready to publish directly. The three tiers — timer polling, notification-triggered polling, direct publish — range from zero integration to instant delivery.
\nTwo design decisions make shared poll feel responsive despite being poll-based.
\nCold key auto-poll. When a client tracks a key with version 0 (\"I have no data\") and no other connection on the same node is tracking that key, Centrifugo triggers an immediate backend poll for it — without waiting for the next scheduled cycle. Data arrives within milliseconds. This means the first user to view a post gets its vote count almost instantly, and subsequent users on the same node benefit from the cached state. No additional configuration is needed.
\nReconnect resilience. Centrifugo has always been designed for graceful reconnection — stream subscriptions catch up from history, and the client SDKs handle the entire reconnect lifecycle transparently. Shared poll works the same way. When a client disconnects and reconnects, the SDK automatically replays all tracked keys using the existing signature and sends the last-known version for each key. Centrifugo compares versions and only pushes data that changed while the client was offline — avoiding a full data re-delivery. The getSignature callback is only invoked when the signature actually expires, not on every reconnect — preventing a mass backend request storm when thousands of clients reconnect simultaneously (e.g., after a load balancer restart).
Publisher-restart resilience. Versioned mode relies on monotonic versions — but a publisher process that resets its in-memory counter on restart would emit \"stale\" versions and connected clients would freeze on their last-seen state. To handle this, the publisher attaches a per-process epoch to each publish and refresh response. When Centrifugo sees the epoch change, it treats the channel as fully reset: per-key versions are wiped, current subscribers are unsubscribed with an insufficient-state code, and SDK auto-resubscribe machinery picks up the new epoch and fresh state — typically within milliseconds. The publisher just needs a fresh string at startup (UUID, a timestamp, anything unique per process lifetime). Empty epoch is also valid and means \"pure version comparison, no restart protection\" — fine when the publisher's lifecycle outlives all subscribers.
\nThe polling cycle guarantees an eventually consistent view of your backend data — clients always converge to the latest state. The delivery tiers above only change how fast they converge, not whether they do. Timer polling alone caps latency at the refresh interval; add direct publish or notifications and updates land in milliseconds. Same correctness, different latency.
\nA use case that highlights this well is configuration sync. Track a single key like app_settings with a long refresh interval (say, 30 seconds). All connected clients catch up to the current configuration through the regular poll cycle. When an admin changes a setting, your backend calls shared_poll_publish — every client receives the update instantly. New clients connecting later get the configuration immediately via cold key auto-poll. You get configuration distribution with just one tracked key and a publish call on write — no Kafka, no extra infrastructure.
When a channel tracks thousands of items, polling them all in a single backend request would create periodic load spikes — and Centrifugo has always prioritized protecting backends from thundering herds. Shared poll splits tracked items into batches and spreads them evenly across the polling interval:
\nRefresh cycle (interval=1s, 3000 tracked keys, batch_size=1000)
Clients Centrifugo Backend
─────── ────────── ───────
│ │ │
│ track(keys) │ │
├────────────────────►│ │
│ │ collect all tracked keys │
│ │ split into batches │
│ │ │
│ t=0 │── batch 1 (keys 1-1000) ─────►│
│ │ │
│ t=333ms │── batch 2 (keys 1001-2000) ──►│
│ │ │
│ t=666ms │── batch 3 (keys 2001-3000) ──►│
│ │ │
│ │◄── responses ─────────────────│
│ │ │
│ │ compare versions │
│ │ per client │
│ │ │
│ update(key, data) │ │
│◄────────────────────│ push only changed items │
│ │ │
│ t≈1s │ next cycle starts │
│ │ │
The backend sees steady load rather than a spike every second. A global concurrency limit (default 64) caps the number of simultaneous backend calls across all channels, preventing Centrifugo from overwhelming a shared data source when many channels refresh simultaneously.
\nThe OSS shape we've described works for most deployments. Three patterns start to bite at higher scale, and Centrifugo PRO addresses each:
\nkeep_latest_data keeps the latest value and version per key in memory; new clients tracking a known key get data straight from cache, with no backend call. The wait drops from \"until next cycle\" to milliseconds.keep_latest_data, PRO computes fossil deltas between successive values per key — clients that negotiated delta receive the patch instead of the full payload. The savings scale with payload size and how rarely it fully changes.prev_data for delta compression with no changes on your backend side.The server-side polling inversion makes shared poll work for use cases push-only systems can't serve: live counters over legacy databases, configuration sync from third-party APIs, data feeds from systems your team doesn't own. These never really fit a WebSocket server before.
\nIt depends on Centrifugo being self-hosted. The refresh proxy hits your backend over the local network — single-digit milliseconds, not a public-internet round-trip — so per-key HMAC verification stays cheap on the hot path, and you can authorize via the same internal services you already trust. A cloud real-time provider can't poll your backend on your behalf without you exposing it publicly or running a tunnel.
\nOne practical scaling note: shared poll channels don't use inter-node PUB/SUB. Each node polls the backend independently and delivers to its own connected clients. A cluster running only shared poll needs no Redis or NATS for messaging coordination.
\nShared poll subscriptions are currently experimental — we may adjust configuration, client SDK API, and proxy protocol based on feedback. At this point, only centrifuge-js supports shared poll subscriptions on the client side.
We've published two interactive demos.
\nVotes — live vote results with dynamic tracking as posts scroll into view:
\n\nDrones — real-time geospatial tracking of 500 simulated drones over a San Francisco map using cell-based spatial partitioning:
\n\nRead the full shared poll documentation for configuration reference, proxy protocol details, and backend signature generation examples in six languages.
", "url": "https://centrifugal.dev/blog/2026/05/21/shared-poll-subscriptions", "title": "Shared poll subscriptions: O(unique items) polling with low-latency updates", "summary": "Shared poll subscriptions move polling from clients to Centrifugo. Instead of 10,000 clients each hitting your backend every second, Centrifugo asks once per cycle for the union of items everyone is watching and fans the changes back out.", "date_modified": "2026-05-21T10:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "websocket", "state-sync", "polling" ] }, { "id": "https://centrifugal.dev/blog/2026/03/01/scaling-ai-token-streams-with-centrifugo", "content_html": "In a previous post we showed how to stream LLM responses through Centrifugo — backend receives tokens from an LLM API, publishes them to a channel, browser subscribes and renders text as it arrives.
\nThat post covered the basics. This one looks at what Centrifugo adds beyond just delivering tokens from point A to point B — automatic recovery after disconnects, horizontal scaling, transport flexibility, and multi-tab synchronization backed by a database. We built an interactive playground demo that demonstrates the concepts – you can run it locally and see every feature in action.
\nThe source code is on GitHub. Run it:
\ngit clone https://github.com/centrifugal/examples.git
cd examples/v6/scale-ai
docker compose up --build
Open http://localhost:9000.
\nThe playground simulates an AI token stream without requiring an actual LLM API. You control the token rate, total token count, and other parameters. The backend picks a random AI-related question, generates random words as the \"answer\", and publishes them to Centrifugo — the delivery path is identical to a real LLM integration.
\nThe architecture:
\n ┌─────────────────────┐
│ nginx :9000 │
└──┬───────────────┬──┘
│ │
▼ ▼
┌──────────┐ ┌─────────┐ ┌────────────┐ ┌───────┐
│ postgres │◀──│ backend │──▶│ centrifugo │──▶│ redis │
└──────────┘ │ :5000 │ │ :8000 │ └───────┘
└─────────┘ └────────────┘
Nginx serves the frontend and proxies /api to the backend, /connection and /emulation to Centrifugo. The backend publishes tokens via Centrifugo HTTP API and persists stream state in PostgreSQL. The frontend subscribes to channels using centrifuge-js.
Let's walk through each feature.
\nStart a stream with default settings: 30 tokens/sec, 100 tokens total. Tokens appear in real-time in the output area. The stats bar shows messages and tokens incrementing.
\n\nThis is the simplest case — one token per message, one message per publish call to Centrifugo HTTP API:
\nasync def publish_to_centrifugo(channel: str, data: dict):
await http_client.post(
CENTRIFUGO_API_URL,
json={\"channel\": channel, \"data\": data},
headers={\"X-API-Key\": CENTRIFUGO_API_KEY},
)
On the client side, the subscription receives each token:
\nsubscription.on('publication', (ctx) => {
const msg = ctx.data;
if (msg.text) {
appendText(msg.text + ' ');
}
});
With default settings, 100 tokens = 100 messages. You could achieve this with plain SSE — one endpoint, one event stream. But production AI streaming hits problems that SSE alone doesn't solve: what happens when the connection drops mid-stream? When the user switches networks? When you need to scale beyond one server? When multiple tabs need the same stream? The rest of this post walks through how Centrifugo handles each of these at the infrastructure layer.
\nBefore moving on — try toggling \"Publisher-side aggregation\" in the controls, set it to 5, and start a new stream.
\n\nThe token count still reaches 100, but the message count drops to ~20. The backend buffers N tokens and publishes them as a batch. At 80 tokens/sec with aggregation of 5, you go from 80 to 16 publish calls per second per stream — fewer syscalls at every layer, and the text still flows naturally. This is a general technique, not Centrifugo-specific, but it composes well with Centrifugo's delivery.
\nStart a stream and click \"Simulate Disconnect\" while tokens are flowing. You'll see:
\n[DISCONNECTED] marker — the client called centrifuge.disconnect()[RECONNECTING...] marker — the client calls centrifuge.connect()[RECOVERED] marker — all missed tokens arrive at once, the stream continuesThe recovery detection on the client:
\nsubscription.on('subscribed', (ctx) => {
if (ctx.wasRecovering && ctx.recovered) {
appendMarker(' [RECOVERED] ', 'text-green-400');
}
});
This is Centrifugo's history-based recovery. The channel is configured with history_size: 500 and force_recovery: true. When the client reconnects, Centrifugo sends all publications that were missed during the disconnect. The client SDK handles this transparently — the publication event fires for each missed message in order.
In the playground we call disconnect() and connect() manually to make the gap visible. In a real application you wouldn't do this — Centrifugo SDKs have built-in reconnect with exponential backoff. When the network drops, the client reconnects automatically and recovery happens without any application code involved. The SDK handles the full cycle: detect disconnect, reconnect, re-subscribe, recover missed messages.
Building this from scratch is a real project: reconnect logic, offset tracking on the client, message buffering on the server, a catch-up protocol. With Centrifugo it's a config option and SDK behavior you get out of the box.
\nMobile networks drop, laptops sleep, WiFi switches. Users won't notice a brief gap if the response continues from where it left off.
\nThe playground already runs Centrifugo with Redis as the engine:
\n{
\"engine\": {
\"type\": \"redis\",
\"redis\": {
\"address\": \"redis:6379\"
}
}
}
This is what makes recovery work — channel history is stored in Redis, not in Centrifugo's process memory. If you restart Centrifugo while a stream is active, recovery still works because the history survives in Redis.
\nRedis also decouples the real-time layer from the backend. The backend publishes tokens to Centrifugo via HTTP API, then moves on — it doesn't hold WebSocket connections, doesn't track who's listening, doesn't buffer messages. Centrifugo and Redis handle all of that. This means you can scale each layer independently: add more backend instances to handle more concurrent LLM calls, add more Centrifugo nodes to handle more subscribers. They share state through Redis — the backend publishes to any Centrifugo node, Redis routes messages to the node that holds the subscriber's connection.
\nThe playground uses a single Redis and single Centrifugo instance, but the architecture is ready for horizontal scaling — just add more Centrifugo nodes pointing to the same Redis. In production, Centrifugo supports Redis Cluster and Redis Sentinel for high availability.
\nThe playground has a transport selector: WebSocket, HTTP Streaming, SSE. Try starting a stream with each — the token delivery works identically regardless of the transport underneath. If you switch transport mid-stream, recovery kicks in and delivers missed tokens, but the real point is that the application code doesn't change at all.
\n\nfunction buildTransports(type) {
switch (type) {
case 'websocket':
return [{ transport: 'websocket', endpoint: `${proto}//${host}/connection/websocket` }];
case 'http_stream':
return [{ transport: 'http_stream', endpoint: `${httpProto}//${host}/connection/http_stream` }];
case 'sse':
return [{ transport: 'sse', endpoint: `${httpProto}//${host}/connection/sse` }];
}
}
Why this matters: corporate networks often run TLS-intercepting proxies that terminate HTTPS, inspect traffic, and re-encrypt it. These proxies may not forward the HTTP Upgrade handshake that WebSocket requires — even when the original connection is over TLS. If your only transport is WebSocket, those users silently fail to connect. With Centrifugo, you configure SSE and HTTP-streaming as fallbacks. The client SDK can try transports in order and use the first one that connects.
From the application's perspective, nothing changes — the subscription API, recovery, channel multiplexing all work identically regardless of the underlying transport. The Centrifugo config to enable these:
\n{
\"sse\": { \"enabled\": true },
\"http_stream\": { \"enabled\": true }
}
For SSE and HTTP-streaming, Centrifugo uses an emulation layer for the bidirectional part (subscribe/unsubscribe commands). Nginx needs to proxy the /emulation endpoint in addition to /connection.
A related point worth mentioning: services like OpenAI use SSE for streaming, partly because SSE over HTTP/2 lets multiple streams share a single TCP connection. Centrifugo supports WebSocket over HTTP/2 (RFC 8441), where each WebSocket connection becomes an HTTP/2 stream inside a shared HTTP/2 connection. You get the multiplexing benefits of SSE/HTTP/2 while keeping bidirectional communication and Centrifugo features like recovery.
\nThe features above work within a single tab's lifecycle. But what happens when a user opens a second tab? Or navigates back to a page while a stream is still running? The tab needs to discover the active stream, catch up on tokens it missed, and continue receiving live updates.
\nThe playground demonstrates this. Start a stream in one tab, then open a fresh tab — it automatically picks up the same stream, shows accumulated tokens, and continues with live ones.
\n\nThe backend persists each stream in PostgreSQL — the question, accumulated answer, and status. Only two writes happen: an INSERT when the stream starts and an UPDATE when it finishes (with the complete answer and status='done'). No database writes during streaming — Centrifugo handles real-time delivery.
Discovering streams on page load. When a tab opens, it calls GET /api/stream/active to find the most recent stream. If the stream is still active, the tab joins it. If it's already done, the tab shows the full question and answer from the database.
Discovering streams in real-time. What about a tab that's already open when someone starts a new stream? Every tab subscribes to an ai:notifications channel. The backend publishes a notification there when a stream starts:
await publish_to_centrifugo(\"ai:notifications\", {
\"type\": \"stream_started\",
\"id\": stream_id,
\"channel\": channel,
\"question\": question,
})
When a tab receives this notification, it joins the new stream immediately — no polling, no page reload.
\nSubscribing with catch-up. Every tab — including the one that started the stream — subscribes to the stream's channel with since: {offset: 0, epoch: ''}. This tells Centrifugo to deliver all existing channel history through the normal recovery flow, then continue with live publications — all through the same publication handler, in order:
const opts = { since: { offset: 0, epoch: '' } };
subscription = centrifuge.newSubscription(channel, opts);
subscription.on('publication', (ctx) => {
// History and live publications arrive through the same handler, in order.
processPublication(ctx);
});
No client-side buffering, no deduplication logic. Centrifugo's recovery mechanism handles the sequencing. This also eliminates the race condition between starting the stream and subscribing — even if the backend publishes tokens before the client subscribes, they arrive through history recovery.
\nBuilding this from scratch means implementing ordered pub/sub with offset tracking, fan-out to multiple subscribers, and a protocol to distinguish history replay from live delivery. Centrifugo provides all of these as built-in primitives.
\nThis is a realistic pattern for combining a database with real-time delivery:
\nA new tab can appear at any point during a stream, catch up seamlessly, and continue with live tokens. This works because Centrifugo's core operation is fan-out: publish once, deliver to all subscribers on the channel. No extra work on the backend, no tracking of which tabs or devices are connected.
\nPractical cases: a user starts a conversation on desktop, picks up the phone, the response keeps streaming. Or a customer support scenario where an AI suggests responses and multiple agents see the same stream. These are all just N subscribers on one channel. And since Centrifugo supports channel multiplexing, the AI stream, a notifications feed, and presence indicators can all share a single WebSocket connection — each additional subscription is virtually free.
\nEach of these features solves a real problem in AI streaming:
\nNone of these are AI-specific features — they're general real-time infrastructure capabilities that happen to be exactly what AI token streaming needs. Implementing them from scratch is significant engineering work: retry logic, offset tracking, message buffering, transport negotiation, fan-out routing. Centrifugo provides them as a single infrastructure layer.
", "url": "https://centrifugal.dev/blog/2026/03/01/scaling-ai-token-streams-with-centrifugo", "title": "Scaling AI token streams with Centrifugo", "summary": "An interactive demo playground demonstrating what Centrifugo brings to AI token streaming — channel multiplexing, recovery, multi-tab sync, transport fallbacks, and horizontal scaling with Redis.", "date_modified": "2026-03-01T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "ai", "streaming", "tutorial", "websocket", "sse" ] }, { "id": "https://centrifugal.dev/blog/2025/10/14/server-side-publication-filtering-by-tags", "content_html": "Real-time applications often face the challenge of delivering relevant content to subscribers while minimizing bandwidth usage and client-side processing overhead. Recently introduced publication filtering by tags in Centrifugo OSS and Centrifugo PRO addresses this challenge from a new side – by allowing clients to subscribe to channels with server-side filters, ensuring that only publications with matching tags are delivered to subscribers.
\n![](\"https://centrifugal.dev/img/publication_filtering.png\")
\nThis feature may significantly help with bandwidth optimization for real-time messaging applications, particularly in scenarios where clients would otherwise receive and discard a significant portion of messages in a channel anyway. Not only network costs may be reduced in this case, but also processing overhead which leads to a faster battery drain on mobile devices.
\nDuring last months we observed the increased interest in this feature from Centrifugo users, so it was eventually implemented in Centrifugo v6.4.0. In this blog post we will discuss the design goals, implementation decisions, and performance benchmarks that led to the final solution.
\nWhen designing publication filtering for Centrifugo, we established several key principles:
\nThe filtering mechanism must be zero-allocation during evaluation because it operates in the hot path during broadcasts to many subscribers. Any memory allocations during filtering would significantly impact performance and increase garbage collection pressure.
\nWhile for most applications it may be fine, the predictability of filtering overhead decreases if evaluation allocates. We will see in the benchmarks below how many allocations may be caused by a single publication in channel with 10k subscribers.
\nFilters must be easy to serialize/deserialize to/from Protobuf and be fully JSON compatible, ensuring seamless integration with existing client SDKs and Centrifugo protocol.
\nThe filtering system should be easily constructible programmatically, allowing developers to build dynamic filters based on application conditions without the need in string formatting and templating.
\nThe implementation should remain simple enough and avoid complexity that could limit adoption or introduce security vulnerabilities. The filtering system should only filter based on data that subscribers can already see in publications, ensuring no security boundaries are crossed.
\nCentrifugo ensures permissions on channel level, filters are not adding a new layer of data protection. Subscriber in channel is able to read all the publications in that stream – filters do not add any permission functionality here.
\nTo implement publication filtering, we decided to use tags associated with each publication. Tags are key-value pairs that can be attached to publication:
\nmessage Publication {
...
bytes data = 4; // Data contains publication payload.
map<string, string> tags = 7; // Optional tags associated with publication.
...
}
It's important that tags are already part of the client protocol. Each channel subscriber has access to data and tags of Publication already. So we do not introduce any new security boundaries here.
For examples in this post, let's consider scenario where a channel represents event stream with football match updates. Stream may look like this:
\n// Publication 1
{
\"data\": {
\"minute\": \"23.27\",
\"event_type\": \"possession_change\",
\"event_data\": {
\"team\": \"Real Madrid\",
\"tackler\": \"Arda Guler\"
}
},
\"tags\": {
\"event_type\": \"possession_change\"
}
}
// Publication 2
{
\"data\": {
\"minute\": \"23.30\",
\"event_type\": \"goal\",
\"event_data\": {
\"team\": \"Real Madrid\",
\"scorer\": \"Kilian Mbappe\",
\"assistant\": \"Arda Guler\"
}
},
\"tags\": {
\"event_type\": \"goal\"
}
}
// Publication 3
{
\"data\": {
\"minute\": \"24.10\",
\"event_type\": \"shot\",
\"event_data\": {
\"team\": \"Bayern Munich\",
\"shooter\": \"Harry Kane\",
\"xG\": \"0.85\",
\"outcome\": \"saved\"
}
},
\"tags\": {
\"event_type\": \"shot\",
\"xG\": \"0.85\"
}
}
We can suppose that user does not need all the match events. Some basic filtering example may be: user is only interested in \"goal\" type events.
Initially, we considered using Google's Common Expression Language (CEL) for filtering, which would have provided a familiar and powerful expression syntax. We already use CEL in other parts of Centrifugo, so it seemed a very natural decision initially. For the scenario above the filter could look like:
\ntags[\"event_type\"] == \"goal\"
Seems simple and straightforward, right? True, and we were quite enthusiastic about using CEL, but the devil is in the details. Turned out for this specific part of Centrifugo it was not the best choice.
\nFirst thing to mention, on every subscription we need to compile the CEL expression to a program. Compilation is not a very cheap operation. Here is how compilation may look like with the cel-go library:
func buildCELProgram(expr string) (cel.Program, error) {
env, err := cel.NewEnv(
cel.Variable(\"tags\", cel.MapType(cel.StringType, cel.StringType)),
)
if err != nil { return nil, err }
ast, issues := env.Compile(expr)
if issues != nil && issues.Err() != nil { return nil, issues.Err() }
if ast.OutputType() != cel.BoolType { return nil, errors.New(\"expected bool output type\") }
return env.Program(ast)
}
// Build and use the program against tags.
prg, err := buildCELProgram(`tags[\"event_type\"] == \"goal\"`)
// Handle err.
activation := map[string]any{\"tags\": tags}
out, _, err := sub.Eval(activation)
// Handle err.
if out == types.True {
// Match!
}
An environment made by cel.NewEnv call may be shared between subscriptions, but parsing expression and checking AST must be done per each subscription. We will see the overhead for this below in benchmarks.
For a simple expression like tags[\"event_type\"] == \"goal\":
| Benchmark Name | Iterations | ns/op | B/op | allocs/op |
|---|---|---|---|---|
| CEL | 41,606 | 28,581 | 23,178 | 326 |
Or with a more complex expression like int(tags[\"count\"]) > 42 && double(tags[\"price\"]) >= 99.5 && tags[\"ticker\"].contains(\"GOO\"):
| Benchmark Name | Iterations | ns/op | B/op | allocs/op |
|---|---|---|---|---|
| CEL | 13,784 | 86,623 | 67,560 | 1,060 |
Which is 3x more.
\nSubscription requests are often frequent operation in Centrifugo, so seeing these numbers is of course unfortunate. Possible optimization here could be caching the same expressions and re-using CEL programs, adds some complexity – but feasible.
\nMore importantly though. While compiled CEL expressions are rather fast to evaluate – evaluations of even simple CEL expressions still come with memory allocations. Memory allocations directly add CPU overhead and also create more load on Garbage Collector (GC). These allocations may not be a huge problem in other parts, but during publication broadcast process it's a very unpredictable performance overhead.
\nIn Centrifugo broadcast process prepares a Publication once and then just adds it to each subscriber queue with minimal allocations during this process. Any allocations per each subscriber in that place in channels with many subscribers can be a performance killer. Another place where we would like to not sacrifice the performance is automatic Publication recovery Centrifugo feature. Recovery is already not a very cheap process to do, so any additional overhead caused by filtering is not a good thing.
\nLet's look at evaluating CEL program with expression like tags[\"event_type\"] == \"goal\":
| Benchmark Name | Iterations | ns/op | B/op | allocs/op |
|---|---|---|---|---|
| CEL | 13,632,322 | 79.70 | 32 | 2 |
| Benchmark Name | Iterations | ns/op | B/op | allocs/op |
|---|---|---|---|---|
| CEL | 1,490 | 791,854 | 320,147 | 20,000 |
Or with more complex expression like int(tags[\"count\"]) > 42 && double(tags[\"price\"]) >= 99.5 && tags[\"ticker\"].contains(\"GOO\"):
| Benchmark Name | Iterations | ns/op | B/op | allocs/op |
|---|---|---|---|---|
| CEL | 3,579,276 | 333.5 | 104 | 7 |
| Benchmark Name | Iterations | ns/op | B/op | allocs/op |
|---|---|---|---|---|
| CEL | 354 | 3,366,629 | 1,040,445 | 70,000 |
One more concern against the design goals above is that CEL expressions do not offer programmatic construction. It's just a string. This, while OK for other cases like server-side configurations, does not fit well client applications where filter often must be built programmatically from the application state.
\nMostly due to this couple of reasons we decided to implement a custom filtering system based on a tree of nodes with logical and comparison operators. This approach is not as flexible as CEL, but it covers most common filtering use cases while providing zero-allocation performance during evaluation and programmatic construction capabilities. Let's look at the chosen design more closely.
\nThe filtering system in Centrifugo is based on a tree structure using FilterNode object defined in the client protocol Protobuf schema:
message FilterNode {
// Operation type:
// - If not set or empty → leaf node (comparison)
// - \"and\" → logical AND of child nodes
// - \"or\" → logical OR of child nodes
// - \"not\" → logical NOT of single child
string op = 1;
// Key for comparison (leaf nodes only)
string key = 2;
// Comparison operator for leaf nodes
// \"eq\", \"neq\", \"in\", \"nin\", \"ex\", \"nex\",
// \"sw\", \"ew\", \"ct\", \"gt\", \"gte\", \"lt\", \"lte\"
string cmp = 3;
// Single value for most comparisons
string val = 4;
// Multiple values for set operations
repeated string vals = 5;
// Child nodes for logical operations
repeated FilterNode nodes = 6;
}
This structure may be passed in subscribe request to Centrifugo, it's already part of Protobuf schema, so no additional parsing is required on server side to build the filter tree since it comes ready – only a very fast zero-allocation validation step is performed.
\nIt's also fully compatible with Centrifugo optimized JSON serialization (as a general rule, we avoid using enums and oneof in the Protobuf schema due to that).
The filter is attached to a Subscription and then applied to tags in every channel Publication in a channel during broadcast.
Filter supports a comprehensive set of comparison and logical operators.
\nMore complex filtering with nested conditions may be built using logical operations: and, or, not.
\nAnd the implementation of evaluation is a simple recursive function with zero allocations during evaluation. Here is a simplified version of it (with only one logical operator and a couple of comparisons, full code with all the above's features may be found on Github)
\nfunc Match(f *FilterNode, tags map[string]string) bool {
switch f.Op {
case OpLeaf:
val, ok := tags[f.Key]
switch f.Cmp {
case CompareEQ:
return ok && val == f.Val
case CompareExists:
return ok
}
case OpAnd:
for _, child := range f.Nodes {
if !Match(child, tags) {
return false
}
}
return true
}
return false
}
Overall, this structure allows rather powerful filtering capabilities while maintaining zero-allocation evaluation performance in hot path.
\nThis is how filtering of football match events by event_type tag may look like with FilterNode structure:
const tagsFilter = {
key: \"event_type\",
cmp: \"eq\",
val: \"goal\"
};
const sub = centrifuge.newSubscription(\"match_events:match_id\", {
tagsFilter: basicFilter
});
More complex filters may be built using logical operators:
\nconst tagsFilter = {
op: \"or\",
nodes: [
{ key: \"event_type\", cmp: \"eq\", val: \"goal\" },
{
op: \"and\",
nodes: [
{ key: \"event_type\", cmp: \"eq\", val: \"shot\" },
{ key: \"xG\", cmp: \"gte\", val: \"0.8\" }
]
}
]
};
const sub = centrifuge.newSubscription(\"match_events:match_id\", {
tagsFilter: tagsFilter
});
It's possible to build filters programmatically using helper functions to ensure type safety and reduce boilerplate. For example, for JavaScript we may have:
\nconst Filter = {
// Comparison operators.
eq: (key, val) => ({ key, cmp: \"eq\", val }),
neq: (key, val) => ({ key, cmp: \"neq\", val }),
in: (key, vals) => ({ key, cmp: \"in\", vals }),
nin: (key, vals) => ({ key, cmp: \"nin\", vals }),
exists: (key) => ({ key, cmp: \"ex\" }),
notExists: (key) => ({ key, cmp: \"nex\" }),
startsWith: (key, val) => ({ key, cmp: \"sw\", val }),
endsWith: (key, val) => ({ key, cmp: \"ew\", val }),
contains: (key, val) => ({ key, cmp: \"ct\", val }),
gt: (key, val) => ({ key, cmp: \"gt\", val }),
gte: (key, val) => ({ key, cmp: \"gte\", val }),
lt: (key, val) => ({ key, cmp: \"lt\", val }),
lte: (key, val) => ({ key, cmp: \"lte\", val }),
// Logical operators.
and: (...nodes) => ({ op: \"and\", nodes }),
or: (...nodes) => ({ op: \"or\", nodes }),
not: (node) => ({ op: \"not\", nodes: [node] })
};
// Same filter using helper.
const tagsFilter = Filter.or(
Filter.eq(\"event_type\", \"goal\"),
Filter.and(
Filter.eq(\"event_type\", \"shot\"),
Filter.gte(\"xG\", \"0.8\")
)
);
const sub = centrifuge.newSubscription(\"match_events:match_id\", {
tagsFilter: tagsFilter
});
Someone can go further and build a string expression parser to build FilterNode tree from string representation similar to CEL, but we did not see a strong need for that so far – leaving as an exercise for the developers.
\nWith the described approach it seems that we found a good balance between flexibility, performance, and simplicity – meeting the design goals.
\nNow let's see at some benchmarks we did to compare this custom approach with CEL.
\nWe ran a series of benchmarks comparing FilterNode against CEL. The tests covered both simple expression and more complex expression used earlier, measuring three scenarios:
\ntags[\"event_type\"] == \"goal\"
| Benchmark Name | Iterations | ns/op | B/op | allocs/op |
|---|---|---|---|---|
| FilterNode | 11,606,757 | 103.5 | 328 | 3 |
| CEL | 41,605 | 28,683 | 23,164 | 326 |
| Benchmark Name | Iterations | ns/op | B/op | allocs/op |
|---|---|---|---|---|
| FilterNode | 102,141,284 | 11.69 | 0 | 0 |
| CEL | 14,766,996 | 78.96 | 32 | 2 |
| Benchmark Name | Iterations | ns/op | B/op | allocs/op |
|---|---|---|---|---|
| FilterNode | 10,000 | 112,080 | 0 | 0 |
| CEL | 1,480 | 794,553 | 320,135 | 20,000 |
Analysis:
\nFilterNode compiles nearly 280x faster than CEL – which is obvious because it's just a matter of several struct creations. Once compiled, executes about 7x faster per evaluation with zero allocations. At scale (10k evals), it is roughly 7x faster and completely allocation-free, whereas CEL incurs significant heap usage.
int(tags[\"count\"]) > 42 && double(tags[\"price\"]) >= 99.5 && tags[\"ticker\"].contains(\"GOO\")
| Benchmark Name | Iterations | ns/op | B/op | allocs/op |
|---|---|---|---|---|
| FilterNode | 5,815,352 | 206.8 | 664 | 5 |
| CEL | 13,548 | 88,699 | 67,502 | 1,060 |
| Benchmark Name | Iterations | ns/op | B/op | allocs/op |
|---|---|---|---|---|
| FilterNode | 11,372,026 | 93.78 | 0 | 0 |
| CEL | 3,609,756 | 333.8 | 104 | 7 |
| Benchmark Name | Iterations | ns/op | B/op | allocs/op |
|---|---|---|---|---|
| FilterNode | 1,290 | 923,829 | 0 | 0 |
| CEL | 356 | 3,350,432 | 1,040,445 | 70,000 |
Analysis:
\nFor more complex expressions, FilterNode is even stronger in comparison. Compilation is ~430x faster than CEL, and evaluation is 3–4x faster. Under load (10k evals), FilterNode is nearly 4x faster and entirely allocation-free, while CEL creates GC pressure.
FilterNode approach delivers order-of-magnitude improvements in both latency and memory efficiency compared to CEL, especially under high-throughput workloads like subscription broadcasts.
\nHere is how it may look like in real life with Centrifugo:
\n\nYou can see a simple browser app which subscribes to a channel with stock price ticks with a filter on ticker tag. After that, client only receives messages matching the filter. You can find the client side source code in Centrifuge Go lib examples.
In this example we're also demonstrating the change of filter on the fly by unsubscribing and subscribing again with a different filter. Also note that offsets of messages are not incremental here because of filtering – client only receives messages matching the filter.
\nFor applications dealing with high-volume channels where clients need only a subset of messages, publication filtering offers a compelling solution that optimizes both network usage and client-side processing overhead. This is particularly beneficial for mobile clients where battery life and data usage are critical.
\nThe custom implementation, while less powerful than expression languages like CEL, provides the exact functionality needed for real-time filtering scenarios while maintaining the performance characteristics essential for high-throughput applications.
\nNote, there was no goal here to say that CEL is bad – no, it's an awesome tool for many use cases including existing usages in Centrifugo. Just for this specific case of publication filtering in Centrifugo we needed something more performance predictable and programmable.
\nThe filtering by tags is available starting from Centrifugo v6.4.0, with the documentation available. For now, it's supported only in Javascript SDK, but we plan to add support in other SDKs too depending on demand.
", "url": "https://centrifugal.dev/blog/2025/10/14/server-side-publication-filtering-by-tags", "title": "Publication filtering by tags - reducing bandwidth with server-side stream filtering", "summary": "Learn how Centrifugo's new publication filtering feature allows clients to subscribe with server-side tag filters, significantly reducing bandwidth usage and client-side processing overhead through efficient zero-allocation filtering.", "date_modified": "2025-10-14T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "performance", "go" ] }, { "id": "https://centrifugal.dev/blog/2025/06/17/streaming-ai-gpt-responses-with-centrifugo", "content_html": "Centrifugo may be used as an efficient and scalable transport for streaming AI responses. In this article, we will stream GPT-3.5 Turbo responses in real-time using Centrifugo temporary channels and Python. We will use OpenAI API to get the answers to user's prompts and stream them to the user using Centrifugo. The user will be able to see the response as it is being generated, similar to how ChatGPT works.
\nHere is a video of the final result:
\n\nThe source code of this example is available on GitHub.
\nIn this example, we will use the following technologies:
\nWe will build the backend using FastAPI - which is a modern web framework for building APIs with Python. It is easy to use, and has great support for asynchronous programming, which is perfect for streaming responses.
\nThe entire backend app is about 70 lines of code only:
\nfrom fastapi import FastAPI
from pydantic import BaseModel
from openai import OpenAI
import httpx
import os
app = FastAPI()
client = OpenAI(api_key=os.getenv(\"OPENAI_API_KEY\"))
CENTRIFUGO_HTTP_API_URL = \"http://centrifugo:8000/api\"
CENTRIFUGO_HTTP_API_KEY = \"secret\"
class Command(BaseModel):
text: str
channel: str
@app.post(\"/api/execute\")
async def api_execute(cmd: Command):
await handle_command(cmd)
return {}
class StreamMessage(BaseModel):
text: str
done: bool
async def handle_command(cmd: Command):
text = cmd.text
channel = cmd.channel
try:
response = client.chat.completions.create(
model=\"gpt-3.5-turbo\",
messages=[{\"role\": \"user\", \"content\": text}],
stream=True,
)
for chunk in response:
token = chunk.choices[0].delta.content or \"\"
if token:
await publish_message(
channel,
StreamMessage(text=token, done=False).model_dump()
)
await publish_message(
channel,
StreamMessage(text=token, done=True).model_dump()
)
except Exception as e:
await publish_message(
channel,
StreamMessage(text=f\"⚠️ Error: {e}\", done=True).model_dump()
)
async def publish_message(channel, stream_message):
payload = {
\"channel\": channel,
\"data\": stream_message
}
headers = {
\"X-API-Key\": f\"{CENTRIFUGO_HTTP_API_KEY}\",
\"Content-Type\": \"application/json\"
}
async with httpx.AsyncClient() as http_client:
await http_client.post(
f\"{CENTRIFUGO_HTTP_API_URL}/publish\", json=payload, headers=headers
)
Let's go through the code step by step:
\nCommand to validate incoming requests with text and channel fields./api/execute that accepts POST requests with a Command payload.handle_command function processes the command, sending the user's text to OpenAI's chat completion API and streaming the response.StreamMessage model to structure the messages sent to Centrifugo.publish_message function sends the streamed messages to the specified Centrifugo channel using its HTTP API.async and await allows the application to handle multiple requests concurrently, making it efficient for streaming responses.The frontend in this example is a single index.html file which draws a chat interface, handles user prompts and connects to Centrifugo to receive answer tokens in real-time.
Here is the code for the frontend (frontend/index.html):
<!DOCTYPE html>
<html lang=\"en\">
<head>
<meta charset=\"UTF-8\" />
<title>Chat with GPT Streaming</title>
<script src=\"https://unpkg.com/centrifuge@5.3.5/dist/centrifuge.js\"></script>
<script src=\"https://cdn.tailwindcss.com\"></script>
<meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\" />
</head>
<body class=\"bg-gradient-to-br from-gray-900 via-black to-gray-900 text-gray-200 min-h-screen flex flex-col items-center justify-start py-6 px-4 text-base\">
<div class=\"w-full max-w-2xl bg-gray-800/40 backdrop-blur-lg border border-gray-700 shadow-2xl rounded-2xl overflow-hidden\">
<div class=\"bg-gradient-to-r from-purple-600 to-indigo-600 text-white px-6 py-4 text-2xl font-bold tracking-wide\">
🔮 Chat with GPT with streaming over Centrifugo
</div>
<div
id=\"chat\"
class=\"h-96 overflow-y-auto p-4 space-y-3 bg-gray-900 scrollbar-thin scrollbar-thumb-purple-500 scrollbar-track-gray-800 transition-colors duration-300\"
></div>
<div class=\"border-t border-gray-700 px-4 py-3 bg-gray-900 flex gap-3\">
<input
id=\"input\"
type=\"text\"
placeholder=\"Type your question...\"
class=\"flex-1 border border-gray-600 bg-gray-800 placeholder-gray-400 text-white rounded-lg px-4 py-2 focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:ring-offset-1 transition-shadow duration-200\"
onkeydown=\"if(event.key === 'Enter') handleSend()\"
/>
<button
onclick=\"handleSend()\"
class=\"bg-gradient-to-r from-indigo-500 to-purple-500 hover:from-purple-500 hover:to-indigo-600 text-white font-medium px-4 py-2 rounded-lg transition-shadow duration-200 shadow-lg hover:shadow-2xl\"
>
Send
</button>
</div>
</div>
<script>
const USER = \"User_\" + Math.floor(Math.random() * 1000);
const BACKEND_URL = \"/api/execute\";
const CENTRIFUGO_WS = \"ws://\" + location.host + \"/connection/websocket\";
const centrifuge = new Centrifuge(CENTRIFUGO_WS);
centrifuge.connect();
const chat = document.getElementById(\"chat\");
const input = document.getElementById(\"input\");
function appendMessage(text, id = null, type = \"user\") {
let el = id ? document.getElementById(id) : null;
if (!el) {
el = document.createElement(\"div\");
el.className = `msg px-3 py-2 rounded-lg max-w-full break-words ${
type === \"user\" ? \"bg-blue-500 text-white self-end ml-auto\" : \"bg-gray-700 text-gray-100\"
}`;
el.id = id || \"\";
chat.appendChild(el);
}
el.innerHTML = text.replace(/\\n/g, '<br>');
chat.scrollTop = chat.scrollHeight;
}
async function handleStreamSubscription(channel, replyId) {
const sub = centrifuge.newSubscription(channel);
let reply = \"\";
sub.on(\"publication\", ctx => {
const msg = ctx.data;
if (msg.text) {
const token = msg.text || \"\";
reply += token;
appendMessage(`GPTBot: ${reply}`, replyId, \"bot\");
}
if (msg.done) {
sub.unsubscribe();
}
});
sub.subscribe();
await sub.ready();
}
async function handleSend() {
const text = input.value.trim();
if (!text) return;
input.value = \"\";
const msgId = crypto.randomUUID();
const channel = `stream_${msgId}`;
appendMessage(`${USER}: ${text}`, null, \"user\");
const cmd = {
text: text,
channel: channel,
};
await handleStreamSubscription(channel, msgId);
await fetch(BACKEND_URL, {
method: \"POST\",
headers: { \"Content-Type\": \"application/json\" },
body: JSON.stringify(cmd)
});
}
</script>
</body>
</html>
The key parts of the code are:
\nappendMessage function appends messages to the chat area, distinguishing between user and bot messages.handleStreamSubscription function subscribes to a temporary channel for the user's prompt. It listens for incoming messages from Centrifugo and appends them to the chat interface in real-time.handleSend function sends the user's prompt to the backend API and initiates the stream subscription for the response.crypto.randomUUID(), which is used to create a temporary channel for streaming the response.As we can see frontend connects to Centrifugo WebSocket endpoint and subscribes to a temporary channel for each user prompt. The backend publishes the response tokens to this channel, and the frontend appends them to the chat interface in real-time.
\nHere we run Centrifugo with a simple configuration. The config.json file for Centrifugo will look like this:
{
\"http_api\": {
\"key\": \"secret\"
},
\"client\": {
\"allowed_origins\": [\"*\"],
\"insecure\": true
},
\"log\": {
\"level\": \"debug\"
}
}
Note, we enabled insecure mode for the client, which allows us to not think about authentication in this example. In a real application, you should use secure connections and proper authentication mechanisms. We are also using a simple HTTP API key \"secret\" for the backend to publish messages to Centrifugo – you of course should use a more secure key in your app.
\nWe will use Nginx as a reverse proxy to serve the frontend and route API requests to the backend. Nginx will also handle static files and provide a simple configuration for serving the application. Here is a Nginx server configuration we used (nginx/default.conf):
server {
listen 80;
location / {
root /usr/share/nginx/html;
index index.html;
try_files $uri $uri/ =404;
}
location /api {
proxy_pass http://backend:5000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
location /connection {
proxy_pass http://centrifugo:8000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection \"upgrade\";
proxy_set_header Host $host;
}
}
Basically it consists of three locations:
\n/ – serves the static files from the frontend directory/api – proxies requests to the backend FastAPI application/connection – proxies requests to Centrifugo for establishing a connection properly proxying WebSocket Upgrade headersFinally, we will combine everything with Docker Compose. The docker-compose.yml file will look like this:
services:
centrifugo:
image: centrifugo/centrifugo:v6
container_name: centrifugo
ports:
- \"8000:8000\"
volumes:
- ./centrifugo:/centrifugo
command: centrifugo -c /centrifugo/config.json
env_file:
- .env
backend:
build: ./backend
container_name: backend
ports:
- \"5000:5000\"
volumes:
- ./backend:/app
env_file:
- .env
depends_on:
- centrifugo
command: uvicorn app:app --host 0.0.0.0 --port 5000 --reload
nginx:
image: nginx:latest
container_name: nginx
ports:
- \"9000:80\"
volumes:
- ./frontend:/usr/share/nginx/html:ro
- ./nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
depends_on:
- backend
- centrifugo
Note, that to test the app with real OpenAI API you need to set your OpenAI API key in the .env file:
OPENAI_API_KEY=\"<YOUR_OPEN_AI_TOKEN>\"
We made Nginx available on port 9000, so once you start the application with:
docker compose up
you can access the frontend at http://localhost:9000.
\nIn this article, we have shown how to stream ChatGPT responses in real-time using Centrifugo as a real-time transport. We used FastAPI for the backend and OpenAI API for generating responses, but it may be easily adapted to other LLMs or backend frameworks. The example is simple and effective, and it can be used as a starting point for building more complex applications that require real-time streaming of AI responses.
\nIn real app don't forget to handle user authentication, including proper authentication of user in Centrifugo. For Centrifugo part see for example JWT auth example in our Grand Chat tutorial.
", "url": "https://centrifugal.dev/blog/2025/06/17/streaming-ai-gpt-responses-with-centrifugo", "title": "Streaming AI responses with Centrifugo", "summary": "Centrifugo is an efficient and scalable transport for streaming AI responses. In this article, we will stream GPT-3.5 Turbo responses in real-time using Centrifugo temporary channels and Python. Simple and effective!", "date_modified": "2025-06-17T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "ai", "python", "tutorial" ] }, { "id": "https://centrifugal.dev/blog/2025/04/28/websocket-real-time-leaderboard", "content_html": "Centrifugo v6.2.0 added a bunch of new asynchronous consumers and new publish API fields such as version and version_epoch. Here, we are covering these changes by a new tutorial.
In the tutorial, we'll build a real-time leaderboard application that updates dynamically as scores change. We'll use Centrifugo for real-time updates, Redis for storing leaderboard data, React for the frontend, and some Python to trigger ranking changes. This is a nice example of how Centrifugo can be used to create interactive, real-time applications with minimal effort.
\n\nOur application will:
\nJump directly to the tutorial source code which may be found on Github. Clone the repo, cd into leaderboard directory and run:
docker compose up
Here's the structure of our project:
\nleaderboard/
├── backend/
│ ├── app.py # Python backend service
│ ├── Dockerfile # Backend Docker configuration
│ ├── requirements.txt # Python dependencies
│ └── lua/
│ └── update_leaderboard.lua # Redis Lua script
├── centrifugo/
│ └── config.json # Centrifugo configuration
├── nginx/
│ └── nginx.conf # Nginx configuration
├── web/
│ ├── public/ # React public assets
│ ├── src/ # React source code
│ ├── Dockerfile # Frontend Docker configuration
│ └── package.json # Frontend dependencies
└── docker-compose.yml # Docker Compose configuration
A docker-compose.yml file is used to combine all the parts together:
version: \"3.8\"
services:
redis:
image: redis:7
ports:
- \"6379:6379\"
centrifugo:
image: centrifugo/centrifugo:v6.2.0
volumes:
- ./centrifugo/config.json:/centrifugo/config.json
command: centrifugo --config=/centrifugo/config.json
ports:
- \"8000:8000\"
depends_on:
redis:
condition: service_started
backend:
build: ./backend
depends_on:
redis:
condition: service_started
centrifugo:
condition: service_started
nginx:
image: nginx:alpine
restart: always
ports:
- 8080:80
volumes:
- ./nginx/nginx.conf:/etc/nginx/conf.d/default.conf
depends_on:
backend:
condition: service_started
web:
condition: service_started
web:
build: ./web
volumes:
- ./web:/app
- /app/node_modules
ports:
- \"3000:3000\"
command: sh -c \"npm install && npm start\"
Here we use Redis to keep leaderboard data because it has a nice ZSET data structure which is famous for its fit for the leaderboard's use case – it's very efficient since data is being managed in-memory, allowing for fast read and write operations.
\nRedis also provides a Lua scripting engine that allows us to perform atomic operations on the data. We use lua scripting actively in Centrifugo Redis integrations, and here we will also utilize the power of it to atomically fill Redis STREAM which will be consumed by Centrifugo.
\nThe backend in this tutorial is very thin and is responsible only for simulating score updates. Here is the whole Python code which triggers ranking updates in Redis:
\nimport time
import random
import redis
def main():
r = redis.Redis(host='redis', port=6379)
with open('lua/update_leaderboard.lua', 'r') as f:
lua_script = f.read()
update_leaderboard = r.register_script(lua_script)
leader_names = [
\"Alice\", \"Bob\", \"Charlie\", \"David\", \"Eve\",
]
while True:
leader = random.choice(leader_names)
increment = random.randint(1, 10)
channel = \"leaderboard\"
update_leaderboard(
keys=[\"leaderboard\", \"leaderboard-state\", \"leaderboard-stream\"],
args=[leader, increment, channel]
)
time.sleep(0.2)
if __name__ == \"__main__\":
main()
This Python script:
\nLua script in backend/lua/update_leaderboard.lua is more interesting:
-- Get or create state hash containing both epoch and version
local leaderboard_key = KEYS[1]
local state_key = KEYS[2]
local stream_key = KEYS[3]
local name = ARGV[1]
local score_inc = tonumber(ARGV[2])
local channel = ARGV[3]
-- Increment leaderboard score
redis.call('ZINCRBY', leaderboard_key, score_inc, name)
local epoch = redis.call(\"HGET\", state_key, \"epoch\")
if not epoch then
local t = redis.call(\"TIME\")
epoch = tostring(t[1])
redis.call(\"HSET\", state_key, \"epoch\", epoch, \"version\", 0)
end
-- Increment version atomically using HINCRBY
local version = redis.call(\"HINCRBY\", state_key, \"version\", 1)
-- Always update TTL regardless of whether state is new or existing
redis.call(\"EXPIRE\", state_key, 86400) -- Set TTL (24 hours, adjust as needed)
-- Get leaderboard data
local members = redis.call('ZREVRANGE', leaderboard_key, 0, -1, 'WITHSCORES')
local leaders = {}
for i = 1, #members, 2 do
table.insert(leaders, { name = members[i], score = tonumber(members[i+1]) })
end
-- Prepare payload for Centrifugo publish API command.
local publish_payload = {
channel = channel,
data = { leaders = leaders },
version = version, -- a tip for Centrifugo about state version
version_epoch = epoch, -- a tip for Centrifugo about state epoch
}
-- Add to stream which is consumed by Centrifugo.
local payload = cjson.encode(publish_payload)
redis.call('XADD', stream_key, 'MAXLEN', '~', 10000, '*',
'method', 'publish', 'payload', payload)
return members
This Lua script:
\n1 and 3 are rather self-explaining. But why do we need to maintain leaderboard state incremental version and its epoch?
\nThe reason is that Redis Stream concurrent consumers working with a consumer group cannot maintain message ordering. So, if you have several Centrifugo instances consuming from the same Redis Stream, they are not able to guarantee that messages are processed in the order they were added to the stream.
\nStarting from Centrifugo v6.2.0, we can use version and version_epoch fields to ensure that clients always receive the most up-to-date state of the leaderboard, even if messages are processed out of order. When Centrifugo receives publications with versions less or equal to those already seen – it skips them, so the client does not receive non-actual data. Of course, this makes sense in cases like the one in this tutorial, where we are publishing the entire state to a channel. An important note is that for version logic to work, we will need to enable Centrifugo history for channels since version information is kept by Centrifugo in the history stream meta information object.
The version is an incremental number, and the logic with it should be straightforward to understand. But why do we need version_epoch? The version_epoch is a string that is used to identify the generation of the data. Since Redis is an in-memory data store, it is possible that the data may be lost if the Redis server is restarted or if the data is evicted from memory. By using version_epoch and tracking its change, Centrifugo avoids a situation where the version counter is restarted and Centrifugo ignores all updates until version reaches the number seen before. Using version_epoch is optional – if it's not passed to Centrifugo, then Centrifugo only looks at the version field to make the decision.
Note that while we are using a single leaderboard in this tutorial, you can extend the Lua script to support multiple leaderboards by adding a leaderboard id/name to leaderboard ZSET and state HASH keys. The Redis STREAM key will stay the same – all updates will go through it, just use different channels in the publication object.
\nNow let's configure Centrifugo to consume the Redis Stream and push updates to connected clients.
\nTo do this, we need Centrifugo to consume the Redis Stream, so centrifugo/config.json may look like this:
{
\"client\": {
\"insecure\": true,
\"allowed_origins\": [\"*\"]
},
\"channel\": {
\"without_namespace\": {
\"history_size\": 1,
\"history_ttl\": \"24h\"
}
},
\"consumers\": [
{
\"enabled\": true,
\"name\": \"leaderboard_redis\",
\"type\": \"redis_stream\",
\"redis_stream\": {
\"address\": \"redis:6379\",
\"streams\": [\"leaderboard-stream\"],
\"consumer_group\": \"centrifugo\",
\"num_workers\": 8
}
}
]
}
This configuration consumes the Redis Stream and also enables insecure WebSocket connections, which is handy for tutorial purposes – so we don't need to think about client authentication and channel permissions here.
\nAnother important thing here is that enabling channel history is required for version logic to work because Centrifugo keeps version data in the history meta information.
\nInitialize a new React application:
\nnpx create-react-app web
Navigate to the web directory and install the dependencies:
\ncd web
npm install centrifuge motion bootstrap
Update web/src/App.js:
import React, { useState, useEffect } from 'react';
import { motion } from 'motion/react';
import { Centrifuge } from 'centrifuge';
import 'bootstrap/dist/css/bootstrap.min.css';
import './App.css';
function App() {
const [state, setState] = useState({
leaders: [],
prevOrder: {},
highlights: {},
});
useEffect(() => {
const centrifuge = new Centrifuge(\"ws://localhost:8000/connection/websocket\");
const sub = centrifuge.newSubscription(\"leaderboard\", {});
sub.on('publication', (message) => {
const data = message.data;
setState(prevState => {
const newHighlights = {};
const newLeaders = data.leaders.map((leader, index) => {
let highlightClass = \"\";
const prevRank = prevState.prevOrder[leader.name];
if (prevRank !== undefined) {
if (prevRank > index) {
highlightClass = \"highlight-up\";
} else if (prevRank < index) {
highlightClass = \"highlight-down\";
}
}
if (highlightClass) {
newHighlights[leader.name] = highlightClass;
}
return leader;
});
const newOrder = {};
newLeaders.forEach((leader, index) => {
newOrder[leader.name] = index;
});
return {
...prevState,
leaders: newLeaders,
prevOrder: newOrder,
highlights: { ...prevState.highlights, ...newHighlights },
};
});
});
centrifuge.connect();
sub.subscribe();
return () => {
sub.unsubscribe();
centrifuge.disconnect();
};
}, []);
return (
<div className=\"container mt-5\">
<div className=\"card\">
<div className=\"card-header\">Real-time Leaderboard with Centrifugo</div>
<div className=\"card-body\">
<table className=\"table table-striped\">
<thead>
<tr>
<th scope=\"col\" className=\"rank-col\">Rank</th>
<th scope=\"col\">Name</th>
<th scope=\"col\">Score</th>
</tr>
</thead>
<tbody>
{state.leaders.map((leader, index) => (
<motion.tr
key={leader.name}
layout
className={state.highlights[leader.name] || ''}
>
<td className=\"rank-col\">{index + 1}</td>
<td>{leader.name}</td>
<td>{leader.score}</td>
</motion.tr>
))}
</tbody>
</table>
</div>
</div>
</div>
);
}
export default App;
This React component:
\nmotion animations to highlight changes in rankingsFile web/src/App.css has some CSS styles to make it look better. Here we skip it for brevity.
Finally, let's add Nginx, which is useful to have a single app endpoint that proxies requests to the React frontend and Centrifugo WebSocket connection:
\nserver {
listen 80;
server_name localhost;
location / {
proxy_pass http://web:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection \"upgrade\";
proxy_set_header Host $host;
}
location /connection/websocket {
proxy_pass http://centrifugo:8000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection \"upgrade\";
proxy_set_header Host $host;
}
}
Now that we have all the components set up, let's run the application:
\ndocker compose up
Once everything is running, you can access the application at http://localhost:8080.
\nWe are sending the full state in every publication now, but only part of the data changes. Centrifugo provides a Fossil delta compression algorithm which can be used to reduce the amount of data sent over the network.
\nFirst, let's look at how our WebSocket session looks now by opening the WebSocket tab in Chrome Dev Tools (right-click and open the image in a new tab to see better):
\n![](\"https://centrifugal.dev/assets/images/leaderboard_no_fossil-ee21b6ed584231de38d4e90080d2e2e3.jpg\")
We see the full content is being sent in every message.
\nLet's add Fossil delta compression. To do this, we need to modify the client-side subscription:
\nconst sub = centrifuge.newSubscription(\"leaderboard\", {
delta: 'fossil',
});
And extend Centrifugo config:
\n{
...
\"channel\": {
\"without_namespace\": {
...
\"allowed_delta_types\": [\"fossil\"],
\"delta_publish\": true
}
},
...
}
Reload the app, and see how WebSocket frames look now:
\n![](\"https://centrifugal.dev/assets/images/leaderboard_fossil-513777bd2e4dd25c05e8f14f0af0ca91.jpg\")
Instead of full payloads, we see deltas with much smaller data size (x2 reduction), and the Centrifugo SDK automatically applies deltas correctly under the hood, so no other changes to the application code are required.
\nLeaderboard data is also a good candidate for Centrifugo cache recovery mode feature. We can instantly and automatically load the latest publication known data upon subscription from Centrifugo stream history.
\nTo do this, we need history to be enabled (we already have) and add a couple of extra options to the configuration:
\n{
...
\"channel\": {
\"without_namespace\": {
...
\"force_recovery\": true,
\"force_recovery_mode\": \"cache\"
}
},
...
}
And one more option (since object) to the client-side subscription to trigger recovery upon the initial subscription:
const sub = centrifuge.newSubscription(\"leaderboard\", {
delta: 'fossil',
since: {}
});
After making this, the latest leaderboard data will be immediately displayed to the user upon subscription without the need for extra synchronization of initial state loading and real-time updates.
\nThis is how it looks from the WebSocket frame perspective:
\n![](\"https://centrifugal.dev/assets/images/leaderboard_cache-e1b4526260ca0f8d720d2be958e67589.jpg\")
I.e., we see that the initial data was sent within the subscribe response. And it's in JSON string format here because we are using delta compression. If we disable delta compression, we will see the initial data just as a regular JSON object.
\nWith Centrifugo, only with a little extra effort you can:
\nIn this tutorial, we've built a real-time leaderboard application using Centrifugo, Redis, and React. This demonstrates how Centrifugo can be used to create interactive, real-time applications with minimal effort.
\nBy leveraging the real-time capabilities of Centrifugo, you can create engaging, interactive applications that provide users with immediate feedback and updates.
\nMoreover, Centrifugo comes with many optimizations to make it the most effective solution for real-time applications. For example, it may use Fossil delta compression to reduce the amount of data sent over the network, and it has a cache recovery mode to ensure that clients can recover from network interruptions without losing updates.
", "url": "https://centrifugal.dev/blog/2025/04/28/websocket-real-time-leaderboard", "title": "Building a real-time WebSocket leaderboard with Centrifugo and Redis", "summary": "In this article, we create a real-time leaderboard using Centrifugo, Redis, React and Python. Here we show the usage of Centrifugo built-in asynchronous consumer from Redis Stream. The post additionally showcases Fossil delta compression and cache recovery mode.", "date_modified": "2025-04-28T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "redis", "react", "python", "tutorial" ] }, { "id": "https://centrifugal.dev/blog/2025/01/16/centrifugo-v6-released", "content_html": "We are excited to announce Centrifugo v6 – a new major release that is now live. This release includes fundamental improvements in the configuration to simplify working with Centrifugo from both user and core development perspectives. It also adds several useful features and enhances observability for both Centrifugo OSS and Centrifugo PRO.
\nFor those who never heard about Centrifugo before, it is a scalable real-time messaging server in a self-hosted environment. Centrifugo allows you to build real-time features in your application, such as chat, notifications, live updates, and more. It's modern, fast, reliable and lightweight. Centrifugo is used by many companies worldwide to power real-time features in their applications. Find out more information in introduction.
\n![](\"https://centrifugal.dev/img/v6_most_wanted.jpg\")
\nIn a recent blog post, we talked about notable Centrifugo v5 milestones. The v5 release was a significant milestone in the Centrifugo project's history, introducing numerous new features and improvements. However, the time for a new major release had come.
\nOver the years, Centrifugo has evolved into a robust platform packed with numerous features. However, as Centrifugo's capabilities expanded, so did the complexity of its configuration. Settings became increasingly dispersed across various parts of the codebase, making them harder to manage and understand. Adding new features required modifying more areas than it should have. With v6, we are addressing this challenge head-on by restructuring the configuration system and rethinking its organization.
\nAdditionally, there were a few areas that required improvement but could not be addressed without breaking changes. A couple of deprecated features were removed in v6.
\nLet's begin our dive into the Centrifugo v6 release with a description of the removed parts.
\nSockJS is a JavaScript library that provides a WebSocket-like object, but under the hood, it uses various transports to establish a connection (falling back to HTTP instead of WebSocket in case of connection issues).
\nThe SockJS transport was deprecated in the Centrifugal ecosystem since the v4 release. We encouraged users to reach out if SockJS was still necessary in blog posts and marked it as deprecated in the documentation. However, nobody reached out during this time.
\nThe SockJS client is poorly maintained these days, with issues not being addressed and some transports becoming outdated.
\nSince Centrifugo v4, we have our own WebSocket emulation layer. Unlike SockJS's HTTP-based fallbacks, our layer:
\nIn Centrifugo v6, SockJS has been removed. Our JavaScript SDK, centrifuge-js, will continue to support the SockJS transport for some time to work with Centrifugo v5, but we plan to remove it from the client SDK eventually.
To enable Centrifugo’s built-in bidirectional emulation, you need to enable HTTP streaming or SSE transports in the server configuration, and then configure centrifuge-js to use those as described in its README:
const transports = [
{
transport: 'websocket',
endpoint: 'ws://localhost:8000/connection/websocket'
},
{
transport: 'http_stream',
endpoint: 'http://localhost:8000/connection/http_stream'
},
{
transport: 'sse',
endpoint: 'http://localhost:8000/connection/sse'
}
];
const centrifuge = new Centrifuge(transports);
centrifuge.connect()
Centrifugo v3 introduced the experimental Tarantool engine, and we were excited about it potentially being a solid alternative to Redis. However, since then, the Tarantool engine hasn’t seen many updates and is now missing some important features, like idempotent publishing and delta compression. These features were added to the memory and Redis engines during the v5 release, but unfortunately, Tarantool was left behind.
\nWe were aware of only two setups where the Tarantool engine was used – and both clients eventually moved away from it with our help. Additionally, our usage statistics do not show any notable adoption of the Tarantool engine.
\nThe reality is that while Tarantool provides some interesting technical advantages over Redis, maintaining proper integration with it and keeping it current is impossible given the resources of Centrifugal Labs. Furthermore, there was no significant support from the Centrifugo community to push its development forward.
\nFor these reasons, we decided to remove Tarantool integration from Centrifugo. All Tarantool-related repositories will be moved to read-only mode. For now Centrifugal Labs will focus on Redis, Redis-compatible brokers, and NATS as the main scalability options for Centrifugo.
\nSometimes, it's necessary to drop some ballast to continue a beautiful journey...
\n![](\"https://centrifugal.dev/img/ballast.jpg\")
\nMeanwhile, our Redis integration has been improved. In Centrifugo v4, we migrated to the redis/rueidis Go library, which enabled the Centrifugo node to achieve better Redis communication throughput. You can find more details in the blog post Improving Centrifugo Redis Engine throughput and allocation efficiency with Rueidis Go library.
\nWe have also put more effort into integrating with Redis-compatible storages, such as DragonflyDB, which may unlock new and interesting capabilities without the need to maintain a separate engine.
\nOver the years, Centrifugo's configuration has been built on the approach initially established in its early versions. At the beginning, the number of configuration options was relatively small and manageable. However, with every new version and feature, the configuration became increasingly difficult to maintain and extend.
\nRefactoring this part is a challenging and not particularly enjoyable process, and it inevitably results in breaking compatibility. Nevertheless, for v6, we decided it was time to make this change.
\nCentrifugo v6 configuration has been completely restructured and now consists of distinct blocks – with all the options grouped together to clearly indicate the layer they correspond to.
\nFor example, there is a client top-level configuration block that contains options related to real-time client connections. To illustrate, let's take the allowed_origins option from Centrifugo v5:
{
\"allowed_origins\": [\"https://example.com\"]
}
It was moved under client section in v6:
{
\"client\": {
\"allowed_origins\": [\"https://example.com\"]
}
}
It is now clear which layer of Centrifugo a given option corresponds to. For instance, the allowed_origins option is related to client connections – not to the server API or the admin web interface.
Let's look at another example. Remember that Centrifugo supports not only JSON configuration files but also YAML and TOML? Let’s look at another example, this time in YAML format:
\nport: 8000
token_hmac_secret_key: XXX
admin_password: XXX
admin_secret: XXX
api_key: XXX
allowed_origins:
- http://localhost:3000
presence: true
namespaces:
- name: ns
presence: true
In v6 becomes:
\nhttp_server:
port: 8000
client:
token:
hmac_secret_key: XXX
allowed_origins:
- http://localhost:3000
admin:
password: XXX
secret: XXX
http_api:
key: XXX
channel:
without_namespace:
presence: true
namespaces:
- name: ns
presence: true
Again – we believe it's much more clear now what part of server each option relates to.
\nWe eliminated the situation where options were scattered across the Centrifugo codebase, often with unclear defaults and a non-obvious process for adding a new option. Now, the configuration is represented by a single Go struct. Config sections are organized into nested structs. Defaults are explicitly defined using struct field definition tags, making them easy to identify. This approach is simple to follow and extend – in most cases, adding a new option is as straightforward as introducing a new field in the struct or a nested struct for more complex functionality. Having all options centralized in a single struct also unlocks new possibilities for working with the configuration, as we will explore below.
\nOne aspect we'd like to highlight is that channel options for channels without any namespace prefix are now defined under the channel -> without_namespace block (as shown in the example above). This change ensures that options for channels without a namespace are not mixed with other Centrifugo options at the same level of configuration. Previously, the way namespace options were organized in the codebase led to several bugs – options for channels without a namespace required separate extraction, which was often overlooked. This issue has now been resolved. Additionally, with new config structure we more explicitly encourage users to adopt channel namespaces as a best practice when working with Centrifugo.
A great feature of Centrifugo is that it warns about unknown options in the configuration file and unknown environment variables at startup. This functionality, which helps users identify configuration mistakes, was already present in previous versions and remains in v6. Now, it also supports keys in deeply nested objects and arrays of objects without requiring excessive copy-paste in the codebase.
\nRe-structuring the configuration also affects how environment variables are constructed to configure Centrifugo. This will require users to update their environment variable configurations. To assist with this, we have added configuration converters to the v6 migration guide and introduced new CLI commands, which should greatly simplify the process (see more details below).
\nAn important aspect of the new Centrifugo v6 configuration is that it uses the same TLS configuration object consistently across the entire system. Whenever you are configuring TLS now, you can expect the same field names, just at different configuration levels. TLS for the HTTP server, Redis client, NATS client, Kafka client, PostgreSQL client, and even mTLS support – all can be configured in a unified way.
\nThe new TLS config object, which was already used in some places in v5, allows certs and keys to be passed in three different ways:
\nSo you can choose the method that is most convenient for you.
\nDue to its self-hosted nature, Centrifugo can provide its users an efficient way to proxy various connection events to the application backend, enabling the backend to respond in a customized manner to control the flow. This feature, called proxy is widely used by Centrifugo users. It allows authenticating connections in a customized manner (when the built-in JWT Centrifugo authentication is not suitable), managing channel subscription permissions and publication validations, refreshing client sessions, and handling RPC calls sent by a client over a bidirectional real-time connection.
\nIn v6, there are a couple of notable improvements in the proxy feature configuration to make it simpler to use.
\nFirst, the separation between granular and non-granular proxy modes has been removed. In other words, there is no need to switch to a granular mode and reconfigure everything – proxies may be added gradually in the way which is more suitable for a real-time feature.
\nThe connect and refresh proxies can now be enabled and configured at the client level, while other types of proxies, which relate to channels, are configured within the channel block and enabled at the channel namespace level. RPC proxy configuration is now defined under a separate rpc section in the configuration.
It is now possible to define default proxies for all event types separately - connect, refresh, subscribe, publish, sub_refresh, etc, – each with its own set of options. Previously, all proxies inherited the same set of options, and only endpoints and timeouts could be customized for specific proxy types. This new flexibility allows you to configure the desired proxy behavior without needing to use named proxy objects in many cases. For example, you can now define connect and refresh proxies with different sets of headers for each. While this behavior seems intuitive, it was previously only achievable by using the granular proxy mode and referencing custom proxies by name. Now, named proxy objects may be avoided in many cases where they were required before. So you can postpone using them until a more granular configuration is really required.
To simplify the process of creating a new configuration file or discovering available options, we added a new CLI command defaultconfig.
The defaultconfig command provides a way to get the configuration file with all defaults for all available configuration options. It will be possible using the command like:
centrifugo defaultconfig -c config.json
centrifugo defaultconfig -c config.yaml
centrifugo defaultconfig -c config.toml
Also, in dry-run mode it will be posted to STDOUT instead of file:
\ncentrifugo defaultconfig -c config.json --dry-run
Finally, it's possible to provide this command a base configuration file - so the result will inherit option values from base file and will extend it with defaults for everything else:
\ncentrifugo defaultconfig -c config.json --dry-run --base existing_config.json
In addition to defaultconfig added defaultenv command which prints all config options as environment vars with default values to STDOUT:
$ centrifugo defaultenv
CENTRIFUGO_HTTP_SERVER_ADDRESS=\"\"
CENTRIFUGO_HTTP_SERVER_PORT=\"8000\"
CENTRIFUGO_ADMIN_ENABLED=false
CENTRIFUGO_ADMIN_EXTERNAL=false
CENTRIFUGO_ADMIN_HANDLER_PREFIX=\"\"
CENTRIFUGO_ADMIN_INSECURE=false
CENTRIFUGO_ADMIN_PASSWORD=\"\"
CENTRIFUGO_ADMIN_SECRET=\"\"
...
Like defaultconfig, it also supports the base config file to inherit values from:
centrifugo defaultenv --base config.json
When using --base, if you additionally provide --base-non-zero-only flag – the output will contain only environment variables for keys which were set to non zero values in the base config file. For example, let's say you have Centrifugo v6 JSON configuration file:
{
\"client\": {
\"allowed_origins\": [\"http://localhost:8000\"]
},
\"engine\": {
\"type\": \"redis\",
\"redis\": {
\"address\": \"redis://localhost:6379\"
}
},
\"admin\": {
\"enabled\": false
}
}
Running:
\ncentrifugo defaultenv --base config.json --base-non-zero-only
– will output only variables set in config file to non zero value (thus admin.enabled skipped also):
CENTRIFUGO_CLIENT_ALLOWED_ORIGINS=\"http://localhost:8000\"
CENTRIFUGO_ENGINE_REDIS_ADDRESS=\"redis://localhost:6379\"
CENTRIFUGO_ENGINE_TYPE=\"redis\"
We already like defaultenv a lot ourselves when combined with grep, to quickly try configuring sth in development and finding out the necessary option:
centrifugo defaultenv | grep \"HMAC\"
Use headers emulation when using centrifuge-js SDK in browser and Centrifugo proxy feature to add a custom header to the request from Centrifugo to the backend.
The WebSocket API in web browsers does not allow setting custom HTTP headers, which makes implementing authentication for WebSocket connections from browsers more challenging.
\nCentrifugo’s JWT authentication provides a robust solution by allowing authentication via a JWT token sent internally in the first client protocol message. However, not everyone prefers to use JWT, so many Centrifugo users have configured a connect proxy to authenticate incoming connections.
\nUnfortunately, in such cases, only cookie-based authentication was available, as web browsers automatically include the Cookie header for WebSocket Upgrade requests to the same domain. Other types of authentication, such as appending a Bearer token in a header, were only possible by passing the token in URL parameters or with initial custom data sent in the connection. While these methods work, they are often inconvenient because the backend cannot easily reuse existing middlewares for authentication.
A useful feature introduced in Centrifugo v6 is called Headers Emulation. This feature is available exclusively in our browser SDK, centrifuge-js (and only makes sense there, as other platforms allow setting headers natively). With this feature, it is now possible to provide a custom headers map in the Centrifuge constructor options. These values are automatically translated into HTTP headers when Centrifugo makes connection proxy requests to the backend. Internally, these custom headers are still passed to Centrifugo via the first client protocol message.
We are not the first offering such a workaround BTW - those familiar with Cloudflare WebSocket API may know that Cloudflare Workers allow setting custom headers for WebSocket connections in a Sec-WebSocket-Protocol header. However, that approach violates WebSocket RFC, and Centrifugo provides a better implementation here moving headers passing to its own client protocol level.
\nHere is an example of how to use centrifuge-js with the headers emulation feature:
const centrifuge = new Centrifuge(
\"wss://example.com/connection/websocket\",
{
\"headers\": {
\"Authorization\": \"Bearer XXX\"
}
}
)
There is also a setter method in SDK to update headers later on.
\nNote, that Centrifugo proxy configuration requires a white list of headers to proxy to the backend, the white list will still be used when working with headers sent in such a way.
\nThis feature simplifies browser-based WebSocket authentication, enabling developers to define custom headers in a familiar and efficient way.
\nAnother feature added in Centrifugo v6 simplifies integrating Centrifugo with Kafka via its asynchronous Kafka consumer. Previously, Centrifugo could integrate with Kafka topics but required a special payload format, where each message in the topic represented a Centrifugo API command. This approach worked well for Kafka topics specifically set up for Centrifugo.
\nWith Centrifugo v6, a new Publication Data Mode has been introduced for Kafka consumers. When this mode is enabled, Centrifugo expects messages in Kafka topics to contain data ready for direct publication, rather than server API commands. It is also possible to use special Kafka headers to specify the channels to which the data should be published.
\nThe primary goal of this mode is to simplify Centrifugo's integration with existing Kafka topics, making it easier to deliver real-time messages to clients without needing to restructure the topic's payload format.
\nAdditionally, since Centrifugo allows configuring an array of asynchronous consumers, it is possible to use Kafka consumers in different modes simultaneously.
\nCentrifugo's engine internally consists of two main components: the Broker and the PresenceManager. During the v5 lifecycle, we introduced the ability to set custom brokers and presence managers for different namespaces in Centrifugo PRO. With the v6 release, this separation is now explicit in the OSS edition as well, allowing the Broker and Presence Manager to be configured independently – see more details in Engines and Scalability documentation chapter.
\nConfiguring the NATS broker as an alternative to Redis is now more straightforward, and adding custom brokers is simpler than before.
\nOne potentially useful application of this flexibility is using separate Redis installations for the Broker and PresenceManager, allowing them to scale independently. Alternatively, you could use NATS for a \"at most once\" broker implementation while keeping Redis for presence management. This provides more flexibility for Centrifugo OSS users to tailor their setups to their specific needs.
\nWe continue to support configuring the Redis engine in Centrifugo v6, as this approach works well for many users and remains the default recommendation. With the new configuration layout, it looks like this:
\nengine:
type: redis
redis:
address: localhost:6379
Several useful metrics have been added to Centrifugo in v6 to improve monitoring and observability.
\nThe first one, centrifugo_client_connections_inflight, shows the number of inflight connections over a specific transport. This is particularly useful when using WebSocket with fallbacks, as it allows you to easily see the percentage of users who are unable to establish a WebSocket connection.
The next metric, centrifugo_command_errors_total, is a counter that tracks API command errors with response code resolution.
Another counter, centrifugo_api_command_errors_total, helps identify which API commands return errors, including Centrifugo-specific error codes.
Asynchronous consumers now have dedicated metrics as well, allowing you to monitor the number of messages consumed by each consumer and the number of processing errors.
\nCentrifugo also provides metrics for the Redis broker PUB/SUB layer, such as the number of errors and the inflight buffered messages in PUB/SUB processor workers. These metrics are invaluable for monitoring the system and gaining a deeper understanding of its state.
\nFor a full description of all available metrics in Centrifugo v6, see the exposed metrics documentation.
\nWe understand how important it is to have a solid starting point for monitoring Centrifugo. That's why we updated our Grafana dashboard to include all the new metrics added during the Centrifugo v5 lifecycle as well as those introduced in v6. The dashboard now features 42 panels (up from 22) for the OSS edition, providing a much better understanding of what's happening with your installation.
\nWe also reconsidered how rates are displayed on the dashboard. Instead of summing up rates per minute, we now display rates per second, which aligns more closely with common practices in the Prometheus ecosystem.
\n![\"Updated](\"https://centrifugal.dev/img/grafana.jpg\")
\nAdditionally, we've expanded the Grand Chat tutorial to include a new chapter on integrating a Prometheus and Grafana stack with the tutorial messenger application.
\nOther improvements introduced in the v6 release include:
\nmessage_size_limit option for WebTransport, which effectively limits the maximum size of individual messages sent through a WebTransport connection..env file with environment variables – can be handy for local development.address option, with support for redis+cluster:// and redis+sentinel:// schemes. Previously, the address option only supported standalone single Redis setups. A key addition is the ability to set different Redis master names for setups reusing the same Sentinel nodes to manage different Redis shards. This approach also simplifies Redis configuration, especially for sharded setups, as each shard can be represented by a separate address string with its own Redis options in the address array. See the updated Engines and Scalability documentation chapter.Node object in many places where it was used only for logging purposes – an awkward legacy now resolved. This refactor also allowed us to utilize the zerolog library for the fastest way to write logs, with strictly typed log values replacing the previously used untyped field maps.Additionally, a significant amount of work has been done on the documentation:
\nFinally, we updated our official Helm chart and the source code of the Grand Chat Tutorial to reflect the changes in v6.
\nCentrifugo PRO v6, as usual, inherits all the changes from the OSS edition. The configuration layout refactoring has also affected some parts of the Centrifugo PRO configuration. Notably, the new defaultconfig and defaultenv CLI commands work seamlessly with Centrifugo PRO as well.
Beyond the configuration layout adjustments, several improvements have been made to Centrifugo PRO features.
\nIn v6, we have taken a step forward with the Centrifugo PRO channel state events feature.
\nWe addressed an edge case where the first occupied event in a channel might not be delivered due to a race condition. Additionally, the processing of the vacated events queue has been made more efficient.
Furthermore, if a channel state proxy is defined in a namespace, it is no longer necessary to explicitly provide an array of events to send. Once the proxy is enabled, Centrifugo PRO will send both occupied and vacated events by default.
That said, we still consider this feature to be in an alpha state.
\nA dedicated PostgreSQL push notifications queue configuration has been added in Centrifugo PRO. This means it is no longer necessary to use the same PostgreSQL instance for the push notifications queue and device management – they can now be separate.
\nIf you are using Centrifugo push notifications solely for broadcasting pushes to known FCM/APNs/HMS tokens, without utilizing the device management API, you can avoid the creation of extra tables in PostgreSQL that Centrifugo would otherwise create for device data storage and related functionality.
\nWe have extended the Grand Chat tutorial to include a new appendix chapter on integrating push notifications with Centrifugo PRO. This tutorial will help you understand how to send Web push notifications to mobile devices using Centrifugo PRO and FCM.
\nWhile the chapter is still under construction, it already includes all the necessary parts combined into a working implementation.
\n\nCentrifugo PRO enhances channel configuration with the Channel Patterns feature. This introduces a flexible way to model channels, similar to how developers configure routes for HTTP request processing in HTTP servers.
\nPreviously, PRO users who wanted to use channel patterns had to completely replace the general namespaces mechanism with patterns. In v6, channel patterns can now be used alongside Centrifugo OSS namespaces.
\nEach channel namespace can include a pattern string option to designate it as a pattern namespace. Namespaces with patterns will only be resolved if the channel matches the defined pattern. This enables the use of patterns for some channels and simple prefix-based namespacing for others, making it much easier to transition from Centrifugo OSS to Centrifugo PRO. Channel namespace configurations can now be updated gradually.
Additionally, features that were previously unavailable for channel patterns, such as automatic personal channel subscriptions, are now accessible to PRO users. However, the channel patterns feature must still be explicitly enabled, as it is not enabled by default. This avoids introducing unintended side effects for setups transitioning to the PRO version, especially those with channels starting with / (used for channels without a namespace).
Centrifugo PRO already supported channel namespace resolution for transport message sent/received metrics. This feature is very useful for setups with many namespaces, as it helps identify which namespaces consume more bandwidth.
\nIn v6, we’ve expanded this capability by providing channel namespace resolution for all Centrifugo metrics related to channels. This enhancement offers deeper insights into how Centrifugo is being used in your setup and enables data-driven business decisions.
\nFor example, you can now segment metrics by channel namespace for metrics such as:
\ncentrifugo_node_messages_sent_count / centrifugo_node_messages_received_countcentrifugo_client_command_duration_secondscentrifugo_client_num_reply_errorsThis extended visibility makes it easier to monitor, analyze, and optimize your Centrifugo setup.
\nCentrifugo PRO's ClickHouse analytics previously enabled building client connection distributions using client name and app version segmentation. This made it possible to understand how many clients from different environments – such as browsers, Android, or iOS devices – were currently connected to your Centrifugo setup. This functionality leveraged the fact that our SDKs pass the SDK name to the server and provide a way to redefine it.
\nIn Centrifugo v6 PRO, this information is now exposed directly as part of Prometheus metrics through the centrifugo_client_connections_inflight gauge. To avoid cardinality issues, Centrifugo requires explicit configuration of registered client names and versions.
Additionally, Centrifugo v6 PRO introduces the centrifugo_client_subscriptions_inflight metric, which includes both client name and channel namespace resolution. These metrics offer valuable insights into current and historical Centrifugo usage.
While these metrics are extremely useful, the ClickHouse analytics feature still provides a deeper level of resolution for individual connections and channel subscriptions. However, the use of ClickHouse can now be deferred if these new metrics provide sufficient data to address your needs.
\nMoreover, the metrics mentioned above, including channel namespace resolution and inflight connections/subscriptions, are now integrated into Centrifugo's official Grafana dashboard.
\nThe next improvement for Centrifugo PRO observability is integration with Sentry.
\nWith just a couple of lines in the configuration, you can enable this feature:
\n{
...
\"sentry\": {
\"enabled\": true,
\"dsn\": \"your-project-public-dsn\"
}
}
– and you will see Centrifugo PRO errors collected by your self-hosted or cloud Sentry installation.
\n![](\"https://centrifugal.dev/img/sentry.jpg\")
\nSome Centrifugo users have Redis setups configured with replication. This might be a Redis Sentinel-based primary-replica setup or a Redis Cluster where each shard consists of a primary and several replicas.
\nCentrifugo PRO v6 enables leveraging existing replicas for specific functionalities:
\nThese enhancements extend Centrifugo's scalability options and can help users optimize resource usage, potentially allowing them to maintain high performance on lower infrastructure resources.
\nAnother feature now available in Centrifugo PRO is support for sharded PUB/SUB in Redis Cluster. Sharded PUB/SUB was introduced in Redis 7.0 to address the scalability challenges of PUB/SUB in Redis Cluster. In a normal PUB/SUB setup, all publications are propagated to all nodes in the Redis Cluster, reducing throughput as more Redis nodes are added. The utilization of Redis shards is usually unequal when using PUB/SUB in Redis Cluster as subscriptions land to one of the shards. With sharded PUB/SUB, the channel keyspace is divided into slots (similar to normal keys), and PUB/SUB operations are distributed across Redis Cluster nodes based on channel names.
\nWhen using Centrifugo PRO with the sharded PUB/SUB feature, several important considerations must be kept in mind. This feature changes how Centrifugo constructs keys and channel names in Redis compared to the standard non-sharded setup. Specifically, Centrifugo divides the channel space into a configurable number of \"partitions,\" typically 64 to 128 (though this can be adjusted based on your needs).
\nPartitioning ensures compatibility with Redis Cluster's slot system while maintaining a manageable number of connections between Centrifugo and Redis. Each partition uses a dedicated connection for PUB/SUB communication with the Redis Cluster.
\nWithout partitioning, each Centrifugo node could potentially create up to 16,384 connections to the Redis Cluster—one for each cluster slot—which is impractical. The partitioning strategy avoids this issue and ensures efficient and scalable communication.
\nGiven implementation details mentioned, we recommend using Redis sharded PUB/SUB only if:
\nSwitching to sharded PUB/SUB mode, despite the changes in keys and channel names in Redis, can significantly improve your system's ability to handle larger workloads.
\nIf you reach the scalability limits of a single Redis Cluster with sharded PUB/SUB, you can scale further by adding additional, isolated Redis Clusters. Centrifugo can be configured to use multiple clusters, enabling scaling similar to its consistent sharding mechanism over isolated single Redis instances. In this setup, sharding occurs across multiple Redis Clusters.
\n![\"redis\"](\"https://centrifugal.dev/assets/images/redis_arch-2eed8366b3d81263ea9ec2201512f964.png\")
For those who are not using Tarantool or SockJS, migrating to Centrifugo v6 is primarily a matter of updating the Centrifugo server configuration.
\nWe understand that, given the nature and number of configuration changes, this may not be straightforward. To simplify the migration process, we've prepared an automatic configuration migration tool (which supports both file and environment configuration migration). You can find more details in the Centrifugo v6 migration guide.
\nThe client protocol, server API protocol, and proxy event protocol have remained unchanged. Therefore, after running Centrifugo v6 with a properly updated configuration, you can expect zero issues (well, zero new issues) with your existing integrations. If you have – let us know. If you need assistance with the migration process or have any other Centrifugo-related questions, check out our community channels for support.
\nWe hope you enjoy the new Centrifugo! It's cleaner, simpler to extend, more developer-friendly, and comes with useful features. As always, let the Centrifugal force be with you 🖲
", "url": "https://centrifugal.dev/blog/2025/01/16/centrifugo-v6-released", "title": "Centrifugo v6 released", "summary": "We are excited to tell the world about Centrifugo v6 – a new major release, which is now live. This release contains fundamental changes in the configuration and adds several useful features and more observability to Centrifugo OSS and Centrifugo PRO.", "date_modified": "2025-01-16T00:00:00.000Z", "author": { "name": "Centrifugal team" }, "tags": [ "centrifugo", "release" ] }, { "id": "https://centrifugal.dev/blog/2024/12/23/centrifugo-v5-milestones", "content_html": "While we concentrate our main efforts on Centrifugo v6 development, let's recap the most notable features and improvements introduced during Centrifugo v5 lifecycle.
\nWe are excited about how many things done in 2024 we can be proud of. Also, we are grateful to our community for the feedback, support and love towards the project we see.
\nLet's start remembering what had happened with the project over the year.
\nWe launched the Grand Tutorial, an official step-by-step guide how to build a complex real-time application with Centrifugo. This comprehensive tutorial walks developers through the entire process, implementing client side, server side and Centrifugo integration. Moreover, it gives answers to robust message delivery and provides performance estimation you can expect to achieve with Centrifugo.
\n\nWhile working on the Grand Chat Tutorial, we realized that Centrifugo could natively consume external queues to simplify integration with the existing backends. Thus, at some point of v5 life cycle built-in asynchronous consumers were introduced. At this point we have two types of them:
\nYou can find examples how both can be used in Broadcast using transactional outbox and CDC chapter of the tutorial.
\nIn v6 we continue improving asynchronous consumers and adding a new mode for Kafka consumer – more details very soon! During v6 life cycle we can add more built-in integrations with popular queue systems.
\nThe addition of idempotency_key in publish and broadcast server API methods provides an effective way to prevent duplicate messages and ensure consistency in a distributed system as it gives a way to safely make publication retries upon network failures.
Centrifugo v5.1.0 introduced an experimental support for gRPC proxy subscription streams. This feature allows developers to send individual stream of events to the client using gRPC-streaming based proxies (bidirectional or unidirectional), providing a way to integrate with third-party existing streams using GRPC streaming between Centrifugo and backend.
\nTechnically, this is quite an interesting concept – Centrifugo can multiplex different streams within a single WebSocket connection (or any other supported transport like SSE or HTTP-streaming), and the communication between Centrifugo and the backend is multiplexed within a single HTTP/2 connection (used by GRPC internally).
\n![](\"https://centrifugal.dev/assets/images/on_demand_stream_connections-7cc823b5765cd558dba320f7f7bfa439.png\")
This feature makes it possible to integrate with external streams in munutes. If you've ever heard about WebSocketd – then the idea is very similar, but over the network! With multiplexing, authentication, authorization, scalabilty Centrifugo features coming with the concept out-of-the-box.
\nSee also Stream logs from Loki to browser with Centrifugo Websocket-to-GRPC subscriptions relevant blog post.
\nDelta compression based on Fossil algorithm was introduced to minimize data transfer overhead in channels, optimizing performance and bandwidth usage during real-time message delivery. The feature is quite unique for self-hosted real-time messaging systems. And we'd like to note that now we not only support delta compression in JavaScript SDK, but recently the community contributed support for it to our Java and Python SDKs. And there is an ongoing PR to Swift SDK.
\n![\"delta](\"https://centrifugal.dev/assets/images/delta_abstract-9104c3b2e3b81831daecf3b400e0d798.png\")
See example where delta compression can be very beneficial in Experimenting with real-time data compression by simulating a football match events\n blog post.
\nThe new cache recovery mode in Centrifugo transforms channels into real-time caches by delivering only the most recent message upon resubscription. This feature has proven highly effective for AzuraCast's \"Now Playing\" functionality, ensuring effective updates for users. With Centrifugo’s cache recovery mode, the system now operates more efficiently, significantly reducing the load on the backend.
\nSeveral new metrics were added to provide deeper insights into Centrifugo operations. A couple of examples:
\ncentrifugo_node_pub_sub_lag_seconds: tracks lag in pub/sub processing – may be very important for server monitoring.centrifugo_client_ping_pong_duration_seconds: measures client latency distribution on top of existing ping-pong mechanism with transport type resolution.Metrics would be one of the main focuses of Centrifugo v6, so we will continue working hard on better observability of Centrifugo.
\nThe useful addition in Centrifugo PRO was a possibility to use per-namespace brokers and presence managers. So it's possible to use wildcard subscriptions with Nats in one channel namespace, and benefit from Redis broker features (presence, history cache and automatic recovery) in other channel namespaces.
\npresence_stats performance for channels with a large number of active subscribers, making Redis-backed deployments more efficient.Several improvements were made in client SDKs. Among others, we'd like to highlight the following:
\nWebSocket is closed before the connection is established error preventing reconnect after long offline period, or this one where we worked on the correctness of quick subscribe/unsubscribe scenarios.In v5 release post we introduced Centrifugal Labs – a company around Centrifugo PRO product, the enhanced version of Centrifugo. It was operational since then. During the year we found new PRO customers, and generally passed very important milestones as a company.
\nThe most important for this post is that we continued to improve Centrifugo PRO with new features. Let's recap!
\nThe first thing to mention regarding Centrifugo PRO is compression performance optimizations. We introduced the cache for prepared WebSocket messages to reduce CPU usage and improve compression ratio. This feature was very helpful for our customers who have a lot of data to send over WebSocket connections. Also, delta compression for channels with at most once delivery guarantees is available for Centrifugo PRO users.
\nWe believe that PRO version should provide an enhanced observability as when the business grows it's crucial to have a deep insight into the system. One notable feature is namespace segmentation for in/out message size metrics. This allows identifying namespaces which consume a lot of bandwidth. In Centrifugo v6 we will make a step forward here, stay tuned!
\nOne of the most attractive features of Centrifugo PRO is push notifications support. During the year several notable improvements were made regarding it. Let's recap.
\nTimezone-aware push notifications – Centrifugo PRO can now send push notifications based on device timezone. Or notification templating - allows using variables and substitute them to values based on particular device metadata. Also, notification localizations - for providing translations of push content based on particular device locale. Next, per user device rate limiting - lets app developers be more careful about the number and rate of push notifications on per user device basis.
\nAnd finally, better scalability of push notification device/topic storage by using reads from PostgreSQL replicas.
\nWe published a Channel state events feature preview – to be notified about channel occupied and channel vacated events on the app backend. Note that we do not recommend this feature for production usage yet. While it may seem simple - the implementation is quite complex under the hood, because we try to solve important issues like event ordering, avoiding event race conditions, making sure we survive Centrifugo node restarts, scalability with Redis Cluster. This all requires a careful approach, so we want to step-by-step improve the feature based on the customer feedback. In Centrifugo v6, we make a little step forward with channel state events, will share more details very soon.
\nWe added SSO for admin UI using OpenID connect (OIDC)protocol, this is a very natural feature to have for corporate users who want to use their existing identity provider to authenticate in Centrifugo admin UI instead of custom password management.
\n![](\"https://centrifugal.dev/assets/images/admin_idp_auth-92fc159f4d015d269d1666453dbee5e3.png\")
There were more improvements and features introduced during Centrifugo v5 lifecycle, here we tried to highlight the most notable ones. This year was quite productive for the project, and we hope your work was more productive due to the help of Centrifugo.
\nWe are looking forward to the upcoming year and the release of Centrifugo v6 very soon. Stay tuned for the upcoming announcement where we will introduce the new features and improvements of the new major release.
\nMerry Christmas and Happy New Year everyone! 🎄
", "url": "https://centrifugal.dev/blog/2024/12/23/centrifugo-v5-milestones", "title": "Notable Centrifugo v5 milestones in 2024", "summary": "While we concentrate our main efforts on Centrifugo v6 development, let's recap the most notable features and improvements introduced during Centrifugo v5 lifecycle.", "date_modified": "2024-12-23T00:00:00.000Z", "author": { "name": "Centrifugal team" }, "tags": [ "centrifugo" ] }, { "id": "https://centrifugal.dev/blog/2024/08/19/optimizing-websocket-compression", "content_html": "In a recent blog post, we talked about the Delta Compression feature of the Centrifuge/Centrifugo stack. We continue the topic of real-time data compression here, and in this post, we will show the optimizations for WebSocket compression made recently in collaboration with one of our Centrifugo PRO customers.
\nThe optimizations described here allowed our customer to reduce the transmit bandwidth used for real-time communication by 3x by enabling WebSocket compression, while keeping server CPU and memory utilization at comparable levels. This eventually resulted in notable savings of up to $12,000 per month on their bandwidth bill.
\nThe optimization is now part of our open-source centrifugal/centrifuge library for the Go language and Centrifugo PRO offering. However, the concepts described here can be applied not only to other Go projects but also beyond the Go ecosystem.
\nHuge kudos to the Skin.Club engineering team (and Sergey in particular) for bringing us the case, evaluating the changes, providing graphs from production, and reviewing this post.
WebSocket compression allows reducing the size of messages exchanged between a client and server over a WebSocket connection. Nowadays, it's achieved using the permessage-deflate algorithm (see RFC 7692), which compresses the data before transmission. By reducing the data size, WebSocket compression helps optimize bandwidth usage and may actually lower latency due to the reduced data size over the wire.
Centrifuge library (the core of Centrifugo server) uses Gorilla WebSocket library for its WebSocket transport implementation (actually, we have our own fork with minor modifications, but the core is the same). Gorilla WebSocket supports WebSocket compression by using a Go standard's library compress/flate package. Enabling compression on server side can be done by setting an EnableCompression option of the websocket.Upgrader:
var upgrader = websocket.Upgrader{
EnableCompression: true,
}
Then it's possible to control whether to use compression for a message by using EnableWriteCompression method before you are writing data to the specific connection:
conn.EnableWriteCompression(false)
// Write data.
When paying for bandwidth, minimizing its usage can result in substantial cost savings, even if it leads to higher CPU and memory consumption on the server side. This is why our customer chose to enable WebSocket compression to evaluate its impact on their metrics:
\n![](\"https://centrifugal.dev/img/ws_compression_enabling.jpg\")
\nSorry for the image quality; at that point, we did not know that the case would eventually lead to a blog post. However, what is important to emphasize from these graphs is that when WebSocket compression was enabled:
\nDespite the negative change of CPU and memory utilization, migrating to WebSocket compression was still economically beneficial for our customer. As mentioned earlier, this step alone allowed the company to save up to $12,000 per month on bandwidth costs, while the additional resource usage costs were comparably much lower.
\nFortunately, at Centrifugal Labs, we had prior experience with WebSocket compression performance overhead. This allowed us to suggest some optimizations to mitigate the resource usage degradation. To understand these optimizations, we first need to explore how WebSocket compression works under the hood.
\nIn Go, when you need to compress the WebSocket frame your starting point is compress/flate package from Go standard library. That's what Gorilla WebSocket uses to implement permessage-deflate compression. You can find the implementation in compression.go file.
When compressing frames the user may choose one of the predefined levels of compression - from 1 (BestSpeed) to 9 (BestCompression).
\nOne of the important compression implementation details is that Gorilla WebSocket uses sync.Pool for flate.Writer (separate pool for each level of compression) and flate.Reader types:
var (
flateWriterPools [maxCompressionLevel - minCompressionLevel + 1]sync.Pool
flateReaderPool = sync.Pool{New: func() interface{} {
return flate.NewReader(nil)
}}
)
Having a pool for these objects allows Gorilla WebSocket to reuse existing readers/writers and generally avoid additional allocations. The benefit of having a pool for flate.Writer is enormous, especially when considering how large this type is:
package main
import \"unsafe\"
import \"compress/flate\"
func main() {
var w flate.Writer
println(unsafe.Sizeof(w))
// Output: 656640
}
So it's around 650 KB on its own! For flate.Reader it's not that critical as it's only 16B overhead. Having pools makes a lot of sense and helps Gorilla WebSocket to perform well when compressing data.
Thus when you call Conn.WriteMessage or use Conn.NextWriter API for connections with compression negotiated under the hood Gorilla Websocket library acquires flate.Writer from the proper pool for the time of write operation.
Gorilla WebSocket implements permessage-deflate without context takeover only. Implementing a context takeover would mean attaching a flate.Writer to each connection – which is very memory inefficient as we can see. Though it can be still a viable money-effective bandwidth optimization for WebSocket – especially if the app does not have a lot of concurrent connections.
But below we will describe a scenario in which having just a sync.Pool is not enough to avoid extra allocations.
Let’s delve into the specifics of Centrifuge/Centrifugo. Both the Centrifuge library and the Centrifugo server implement the PUB/SUB pattern for real-time messaging. Clients can establish a WebSocket connection and subscribe to multiple channels, all multiplexed over a single connection. A single channel can have many online subscribers — often thousands. Our APIs enable publishing a message to a channel, ensuring it is delivered to all online subscribers of that channel. Here’s a simple illustration of the PUB/SUB pattern:
\n![\"pub_sub\"](\"https://centrifugal.dev/assets/images/pub_sub-5477abf6fb38219fc0848d9c9c3dc2b1.png\")
When you publish a message to a channel, our WebSocket server puts the message into each subscribed client's individual queue. Client queues are processed concurrently in separate goroutines, and messages are eventually sent to the client over the transport implementation. WebSocket is the most frequently used transport implementation in our case.
\nWith WebSocket compression enabled, broadcasts to a channel with many online subscribers result in many compression operations that compress the same message concurrently. This, in turn, results in a load on sync.Pool, which grows significantly at the point of such a broadcast. Given the size of each flate.Writer object, this is exactly what causes the temporary memory spikes we observed on the graphs above. More allocations also mean more CPU utilization.
In your app, you may observe something like this in the heap profile:
\n![\"img\"](\"https://centrifugal.dev/assets/images/ws_compression_profile-bf2e06a8ae8136796695c14f133d8e8f.jpg\")
That's why, at some point in the past, Gary Burd (author of Gorilla WebSocket) added a PreparedMessage type. This is a helper type that allows caching the constructed WebSocket frame depending on various negotiated connection options (compression used or not, compression level).
\n// PreparedMessage caches on the wire representations of a message payload.
// Use PreparedMessage to efficiently send a message payload to multiple
// connections. PreparedMessage is especially useful when compression is used
// because the CPU and memory expensive compression operation can be executed
// once for a given set of compression options.
type PreparedMessage struct {
messageType int
data []byte
mu sync.Mutex
frames map[prepareKey]*preparedFrame
}
// prepareKey defines a unique set of options to cache prepared frames in PreparedMessage.
type prepareKey struct {
isServer bool
compress bool
compressionLevel int
}
// preparedFrame contains data in wire representation.
type preparedFrame struct {
once sync.Once
data []byte
}
Then there is a method func (*Conn) WritePreparedMessage, which should be used to write data to all connections interested in a message — the proper WebSocket frame will be created once and then automatically re-used. By the way, this is an example of the elegant use of sync.Once.
preparedMessage, _ = websocket.NewPreparedMessage(websocket.TextMessage, data)
_ := conn.WritePreparedMessage(preparedMessage)
This means that if we broadcast the same prepared message to many connections, we remove the excessive load on sync.Pool, just taking a one flate.Writer from the pool instead of many. This way, we avoid large memory spikes due to big size of flate.Writer objects and sync.Pool growth.
For broadcasts, PreparedMessage approach significantly reduces CPU usage by minimizing the need to construct and compress WebSocket frames, especially as the number of concurrent subscribers increases. Additionally, it reduces the allocation of large flate.Writer objects, further optimizing CPU utilization.
Gorilla WebSocket contains benchmarks which compare message broadcast to many connections with enabled compression without and with PreparedMessage usage. Let's run them:
❯ go test -run xxx -bench BenchmarkBroadcast -benchmem
Compression_100_conn-8 198619 ns/op\t 14113 B/op\t 301 allocs/op
CompressionPrepared_100_conn-8 42643 ns/op\t 12320 B/op\t 19 allocs/op
Compression_1000_conn-8 1797432 ns/op\t 123864 B/op\t 3001 allocs/op
CompressionPrepared_1000_conn-8 649506 ns/op\t 11421 B/op\t 19 allocs/op
Compression_10000_conn-8 16506132 ns/op\t 1040709 B/op\t 30007 allocs/op
CompressionPrepared_10000_conn-8 7702265 ns/op\t 11631 B/op\t 21 allocs/op
Benchmarks demonstrate that PreparedMessage significantly reduces memory allocations during broadcasts, with the impact becoming more significant as the number of connections increases.
For Centrifuge/Centrifugo though, we couldn't directly use PreparedMessage in the part of the code responsible for preparing messages for channel broadcasts. This is because doing so would introduce a dependency on a WebSocket-specific type in a layer of code that should remain agnostic to the underlying real-time transport.
To avoid this, we chose not to rely on the PreparedMessage type in the broadcasting preparation layer. Instead, we implemented a cache of PreparedMessage types within the WebSocket transport implementation layer.
We use the data to be sent to the WebSocket connection as a cache key, and the PreparedMessage object as a value. The WebSocket transport implementation checks the cache before constructing PreparedMessage and has a high chance of finding it. The TTL (time-to-live) of each cache entry can be kept very short (something like 1 second should be sufficient). The size of the cache should be comparable to the total size of all different messages being broadcasted concurrently. For most WebSocket applications, a cache size of several megabytes should be more than enough.
We used maypok86/otter cache for our implementation, but there are a lot of other options in Go ecosystem.
\nHere is how we initialize the cache:
\notter.MustBuilder[string, *websocket.PreparedMessage](int(config.CompressionPreparedMessageCacheSize)).
Cost(func(key string, value *websocket.PreparedMessage) uint32 {
return 2 * uint32(len(key))
}).
WithTTL(time.Second).
Build()
And use it at the point where we want to write data to the WebSocket connection on a transport layer:
\n...
if usePreparedMessage {
key := convert.BytesToString(data)
preparedMessage, ok := t.preparedCache.Get(key)
if !ok {
var err error
preparedMessage, err = websocket.NewPreparedMessage(messageType, data)
if err != nil {
return err
}
t.preparedCache.Set(key, preparedMessage)
}
err := t.conn.WritePreparedMessage(preparedMessage)
if err != nil {
return err
}
} else {
...
}
You may wonder why the cache at this level can make a difference compared to simply using conn.WriteMessage. At first glance, the probability of finding a PreparedMessage in the cache during a concurrent broadcast seems similar to getting a flate.Writer from the sync.Pool for reuse. However, that's not entirely true, because the WriteMessage method of Gorilla WebSocket acquires a flate.Writer for the duration of the write operation, which involves a syscall and generally takes much more time than our cache operations. Additionally, two subsequent writes that reuse the flate.Writer will construct the frame from scratch, whereas a cached PreparedMessage allows us to avoid this.
It's important to note that the fact we use a cache is actually a Centrifuge-specific detail to avoid dependency on a type from the WebSocket library in places where we don't want to be tied to specific transport implementations. In your WebSocket app that uses Gorilla WebSocket, you can likely create a PreparedMessage directly and use it when iterating over connections to which you need to broadcast the message. This approach might even lead to more predictable behavior than what we have in the Centrifuge case with a cache. While the cache approach is somewhat probabilistic, the probability is high, and as we'll see below, it works well.
Since we are discussing compression optimizations in the Go ecosystem, it would be wrong not to mention the klauspost/compress library created by Klaus Post. The library is backwards compatible with standard Go packages but provides faster implementations of compression algorithms. Additionally, it offers capabilities beyond those of the standard library.
\nSpecifically, it provides Stateless Compression, which theoretically could help with the use case we described here.
\nHowever, in our initial evaluations of klauspost/compress (switching to its flate implementation and also trying the Stateless Compression feature), we did not observe notable improvements in our specific set of benchmarks (and actually observed regressions in some cases). So, for now, we are sticking with the standard Go library. Nevertheless, it's still a promising direction for research, and we may re-evaluate our findings in the future. Since we use our own fork of Gorilla WebSocket, we can easily switch to klauspost/compress if we prove the benefit.
Alright, it's time to see how enabling PreparedMessage cache helped the customer with a resource usage.
First, let's look at CPU after enabling the cache in production:
\n![](\"https://centrifugal.dev/img/ws_compression_results_cpu.png\")
\nWe see that the average CPU utilization is significantly reduced at peak times, closer to values before enabling the WebSocket compression.
\nSecond, let's look at memory usage:
\n![](\"https://centrifugal.dev/img/ws_compression_results_mem.png\")
\nAs we can see on the graph – the pattern smoothed a lot making the usage on each Centrifugo node much more stable. It's still not the same as it was with compression disabled, we can see minor spikes – but the positive effect of PreparedMessage cache is clear.
Enabling WebSocket compression helped our customer significantly reduce monthly costs, and with PreparedMessage cache optimization, we were able to keep resource usage at a comparable level. So, it was mostly a no-brainer improvement.
\nIf you are a Go developer, our open-source centrifugal/centrifuge library for Go supports WebSocket compression and contains the cache described here — so if you build your app on top of Centrifuge, you just need to enable a couple of options to benefit from it:
\nwebsocketConfig := centrifuge.WebsocketConfig{
Compression: true,
CompressionPreparedMessageCacheSize: 1048576 // 1 MB.
}
If we talk about Go, the technique with prepared message is also applicable to other WebSocket libraries, not just Gorilla WebSocket. For example, with gobwas/ws, it seems possible to construct a prepared compressed WebSocket frame, though it involves slightly more work than with Gorilla WebSocket, which provides a handy PreparedMessage type for this and abstracts the complexity of frame building (based on the set of connection-negotiated options).
That's it for today. In some cases, when working with WebSocket broadcasts, the bandwidth may be reduced even more — check out our post about Delta Compression.
", "url": "https://centrifugal.dev/blog/2024/08/19/optimizing-websocket-compression", "title": "Performance optimizations of WebSocket compression in Go application", "summary": "In this post, we explore how WebSocket compression can optimize bandwidth costs. We also discuss strategies to minimize the CPU and memory overhead associated with the enabled WebSocket compression from the Go ecosystem perspective.", "date_modified": "2024-08-19T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "centrifuge", "websocket", "compression", "performance" ] }, { "id": "https://centrifugal.dev/blog/2024/06/03/real-time-document-state-sync", "content_html": "Centrifugo and its main building block Centrifuge library for Go both provide a way for clients to receive a stream of events in channels using Subscription objects. Also, there is an automatic history recovery feature which allows clients catching up with missed publications after the reconnect to the WebSocket server and restore the state of a real-time component. While the continuity in the stream is not broken clients can avoid re-fetching a state from the main application database – which optimizes a scenario when many real-time connections reconnect all within a short time interval (for example, during a load balancer restart) by reducing the excessive load on the application database.
\nUsually, our users who use recovery features load the document state from the backend, then establish a real-time Subscription to apply changes to the component state coming from the real-time channel. After that the component stays synchronized, even after network issues – due to Centrifugo recovery feature the document state becomes actual again since client catches up the state from the channel history stream.
\nThere are several hidden complexities in the process though and things left for users to implement. We want to address those here.
\nIn the post we assume that you are using a channnel with history configured and recovery on, i.e. using a namespace with history_size, history_ttl and force_recovery on, see more details in History and recovery\n doc chapter.
The first edge case comes from the fact that there is a possible gap in time between initial loading of the state from the main app database and real-time subscription establishment. Some messages could be published in between of state loading and real-time subscription establishment. So there is a chance that due to this gap in time the component will live in the inconsistent state until the next application page reload. For many apps this is not critical at all, or due to message rates happens very rarely. But in this post we will look at the possible approach to avoid such a case.
\nOr imagine a situation when state is loaded, but real-time subscription is delayed due to some temporary error. This increases a gap in time and a chance to miss an intermediary update.
\nCentrifugo channel stream offsets are not binded to the application business models in any way, so it's not possible to initially subscribe to the real-time channel and receive all updates happened since the snapshot of the document loaded from the database. There is a way to solve this though, we will cover it shortly.
\nAnother complexity which is left to the user is the need to react on recovered: false flag provided by the SDK when client can not catch up the state upo re-subscription. This may happen due to channel history retention configuration, or simply because history was lost. In this case our SDKs provide users wasRecovering: true and recovered: false flags, and we suggest re-fetching the document state from the backend in such cases. But while you re-fetch the state you are still receiving real-time updates from the subscription – which leads us to something similar to the problem described above, same race conditions may happen leaving the component in the inconsistent state until reload.
One more possible problem to discuss is a late delivery of real-time messages.
\nWhen you want to reliably stream document changes to Centrifugo (without loosing any update, due to temporary network issues for example) and keep the order of changes (to not occasionally apply property addition and deletion is different order on the client side) your best bet is using transactional outbox or CDC approaches. So that changes in the database are made atomic and there is a guarantee that the update will be soon issued to the real-time channel of Centrifuge-based server or Centrifugo. Usually transactional outbox or CDC can also maintain correct order of event processing, thus correct order of publising to the real-time channel.
\nBut this means that upon loading a real-time component it may receive non-actual real-time updates from the real-time subscription – due to outbox table or CDC processing lag. We need a way for the client side to understand whether the update must be applied to the document state or not. Sometimes it's possible to understand due to the nature of component. Like receiving an update with some identifier which already exists in the object on client side. But what if update contains deletion of some property of object? This will mean that object may rollback to non-actual state, then will receive next real-time updates which will move it to back to the actual state. We want to avoid such modifications leading to temporary state glitches at all. Not all cases allow having idempotent real-time updates.
\nEven when you are not using outbox/CDC you can still hit a situation of late real-time message delivery. Let's suppose you publish messages to Centrifugal channel synchronously over server publish API reducing the chance of having a lag from the outbox/CDC processing. But the lag may still present. Because while message travelling towards subscriber through Centrifugo, subscriber can load a more freshy initial state from the main database and subscribe to the real-time channel.
\nIn this post we will write a RealTimeDocument JavaScript class designed to synchronize a document state. This class handles initial data loading, real-time updates, and state re-synchronization when required. It should solve the problems described above.
The good thing is that this helper class is compact enough to be implemented in any programming language, so you can apply it (or the required part of it) for other languages where we already have real-time SDKs.
\nWe will build the helper on top of several core principles:
\nread committed transactional isolation level (default in many databases, ex. PostgreSQL)We already discussed the approach in our Grand tutorial – but now want to generalize it as a re-usable pattern.
\nAfter writing a RealTimeDocument wrapper we will apply it to a simple example of synchronizing counter increments across multiple devices reliably to demonstrate it works. Eventually we get best from two worlds – leveraging Centrifugo publication cache to avoid excessive load on the backend upon massive reconnect and proper document state in all scenarios.
const subscription = centrifuge.newSubscription('counter', {});
const realTimeDocument = new RealTimeDocument({
subscription, // Wraps Subscription.
load: async (): Promise<{ document: any; version: number }> => {
// Must load the actual document state and version from the database.
// Ex. return { document: result.document, version: result.version };
},
applyUpdate: (currentDocument: any, update: any): any => {
// Must apply update to the document.
// currentDocument.value += update.increment;
// return currentDocument;
},
compareVersion: (currentVersion: number, update: any): number | null => {
// Must compare versions in real-time publication and current doc version.
// const newVersion = publication.data.version;
// return newVersion > currentVersion ? newVersion : null;
},
onChange: (document: any) => {
// Will be called once the document is loaded for the first time and every time
// the document state is updated. This is where application may render things
// based on the document data.
}
});
realTimeDocument.startSync();
To address the gap between state load and real-time subscription establishment the obvious solution which is possible with Centrifugal stack is to make the real-time subscription first, and only after that load the state from the backend. This eliminates the possibility to miss messages. But until the state is loaded we need to buffer real-time publications and then apply them to the loaded state.
\nHere is where the concept of having incremental document version helps – we can collect messages in the buffer, and then apply only those with version greater than current document version. So that the object will have the correct state after the initial load.
\nHere is how we can process real-time publications:
\nthis.#subscription.on('publication', (ctx) => {
if (!this.#isLoaded) {
// Buffer messages until initial state is loaded.
this.#messageBuffer.push(ctx);
return;
}
// Process new messages immediately if initial state is already loaded.
const newVersion = this.#compareVersion(ctx.data, this.#version);
if (newVersion === null) {
// Skip real-time publication, non actual version.
return;
}
this.#document = this.#applyUpdate(this.#document, ctx.data);
this.#version = newVersion;
this.#onChange(this.#document);
}
And we also need to handle subscribed event properly and load the initial document state from the backend:
this.#subscription.on('subscribed', (ctx) => {
if (ctx.wasRecovering) {
if (ctx.recovered) {
// Successfully re-attached to a stream, nothing else to do.
} else {
// Re-syncing due to failed recovery.
this.#reSync();
}
} else {
// Load data for the first time.
this.#loadDocumentApplyBuffered();
}
})
For the initial load this.#loadDocumentApplyBuffered() will be called. Here is how it may look like:
async #loadDocumentApplyBuffered() {
try {
const result = await this.#load();
this.#document = result.document;
this.#version = result.version;
this.#isLoaded = true;
this.#processBufferedMessages();
} catch (error) {
// Retry the loading, in the final snippet it's implemented
// and uses exponential backoff for the retry process.
}
}
After loading the state we prosess buffered real-time publications inside #processBufferedMessages method:
#processBufferedMessages() {
this.#messageBuffer.forEach((msg) => {
const newVersion = this.#compareVersion(msg, this.#version);
if (newVersion) { // Otherwise, skip buffered publication.
this.#document = this.#applyUpdate(this.#document, msg.data);
this.#version = newVersion;
}
});
// Clear the buffer after processing.
this.#messageBuffer = [];
// Only call onChange with final document state.
this.#onChange(this.#document);
}
This way the initial state is loaded correctly. Note also, that version comparisons also help with handling late delivered real-time updates – we now simply skip them inside on('publication') callback.
Let's go back and look how to manage stream continuity loss:
\nif (ctx.recovered) {
// Successfully re-attached to a stream, nothing else to do.
} else {
// Re-syncing due to failed recovery.
this.#reSync();
}
In this case we call #reSync method:
#reSync() {
this.#isLoaded = false; // Reset the flag to collect new messages to the buffer.
this.#messageBuffer = [];
this.#loadDocumentApplyBuffered();
}
It basically clears up the class state and calls #loadDocumentApplyBuffered again – repeating the initial sync procedure.
That's it. Here is a full code for the RealTimeDocument class. Note, it also contains backoff implementation to handle possible error while loading the document state from the backend endpoint.
I've made a POC with Centrifuge library to make sure this works.
\nIn that example I tried to apply RealTimeDocument class to synchronize state of the counter. Periodically timer is incremented on a random value in range [0,9] on the backend and these increments are published to the real-time channel. Note, I could simply publish counter value in every publication over WebSocket – but intentionally decided to send counter increments instead. To make sure nothing is lost during state synchronization so counter value is always correct on the client side.
Here is a demo:
\n\nLet's look at how RealTimeDocument class was used in the example:
const counterContainer = document.getElementById(\"counter\");
const client = new Centrifuge('ws://localhost:8000/connection/websocket', {});
const subscription = client.newSubscription('counter', {});
const realTimeDocument = new RealTimeDocument({
subscription,
load: async () => {
const response = await fetch('/api/counter');
const result = await response.json();
return { document: result.value, version: result.version };
},
applyUpdate: (document, update) => {
document += update.increment
return document
},
compareVersion: (currentVersion, update) => {
const newVersion = update.version;
return newVersion > currentVersion ? newVersion : null;
},
onChange: (document) => {
counterContainer.textContent = document;
},
debug: true,
});
client.connect();
// Note – we can call sync even before connect.
realTimeDocument.startSync();
Things to observe:
\n{\"version\":4823,\"value\":21656} from /api/counter{\"version\":4824,\"increment\":9} over real-time channel/api/counter endpoint not called at all - state fully cought up from Centrifugo history streamIn this post, we walked through a practical implementation of a RealTimeDocument class using Centrifugal stack for the real-time state synchronization to achieve proper eventually consistent state of the document when using real-time updates. We mentioned possible gotchas when trying to sync the state in real-time and described a generic solution to it.
You don't need to always follow the solution here. As I mentioned it's possible that your app does not require handling all these edge cases, or they could be handled in alternative ways – this heavily depends on your app business logic.
\nNote, that with some changes you can make RealTimeDocument class to behave properly and support graceful degradation behaviour. If there are issues with real-time subscription you can still load the document and display it, and then re-sync the state as soon as a real-time system becomes available (successful subscription).
Optimizing data transfer over WebSocket connections can significantly reduce bandwidth costs. Compressing data usually leads to memory and CPU resource usage overhead – but in many cases it worth doing anyway since it positively impacts the final bill from the provider (bandwidth cost reduction overweights resource usage increase).
\nCentrifugo v5.4.0 introduced delta compression feature. But before implementing it we wanted a playground which could demonstrate the potential benefit of using delta compression in Centrifugo channels.
\nThis post outlines our approach to estimating the potential profit from implementing delta compression. It demonstrates the reduction in data transfer using once concrete use case across various configurations, including different Centrifugo protocol formats and the additional use of WebSocket permessage-deflate compression. Although these numbers can vary significantly depending on the data, we believe the results are valuable for providing a general understanding of Centrifugo compression options. This information can help Centrifugo users apply these insights to their use cases.
\n
For a good overview of delta compression topic for the real-time messaging applications I suggest starting with a blog post in Ably engineeiring blog.
\nCentrifugo is very similar to Ably in many aspects (though self-hosted), so everything said in the linked post equally applies to Centrifugo use cases too. Though we have differences in the final implementation, one notable is that we are using Fossil delta algorithm in Centrifugo instead of VCDIFF. The reason over VCDIFF was mainly two factors:
\nThe compactness property is nice because there are no OSS Fossil implementations for Java, Dart and Swift – languages we have SDKs for – so we may have to implement this algorithm in the future ourselves.
\nHaving said this all, let's proceed to the description of experiment we did to understand possible benefits of various compression techniques, and delta compression in particular.
\nIn the experiment, we simulated a football match, sending the entire game state over a WebSocket connection upon every match event. Our compression playground looks like this:
\n\nIt visualizes only the score, but under the hood there are other game changes happen – will be shown below.
\nWe tested various configurations to evaluate the effectiveness of data compression if different cases. In each setup the same game data was sent over the wire. The data then was captured using WireShark with the filter:
\ntcp.srcport == 8000 && websocket
This is how WebSocket packets look in Wireshark when applying a filter mentioned above:
\n![\"wireshark\"](\"https://centrifugal.dev/assets/images/compression_wireshark-b44531ca65e851165c62331d10d661f4.png\")
Bytes captured show the entire overhead from packets in the game channel going from server to client (including TCP/IP overhead).
\nThe source code of the experiment may be found on Github as a Centrifuge library example. You can run it to inspect the exact WebSocket frames in each scenario.
\nTo give reader a general idea about data, we sent 30 publications with the entire football game state, for example here is a first message in a match (2 teams, 11 players):
\n{
\"homeTeam\":{
\"name\":\"Real Madrid\",
\"players\":[
{
\"name\":\"John Doe\"
},
{
\"name\":\"Jane Smith\"
},
{
\"name\":\"Alex Johnson\"
},
{
\"name\":\"Chris Lee\"
},
{
\"name\":\"Pat Kim\"
},
{
\"name\":\"Sam Morgan\"
},
{
\"name\":\"Jamie Brown\"
},
{
\"name\":\"Casey Davis\"
},
{
\"name\":\"Morgan Garcia\"
},
{
\"name\":\"Taylor White\"
},
{
\"name\":\"Jordan Martinez\"
}
]
},
\"awayTeam\":{
\"name\":\"Barcelona\",
\"players\":[
{
\"name\":\"Robin Wilson\"
},
{
\"name\":\"Drew Taylor\",
\"events\":[
{
\"type\":\"RED_CARD\"
}
]
},
{
\"name\":\"Jessie Bailey\"
},
{
\"name\":\"Casey Flores\"
},
{
\"name\":\"Jordan Walker\"
},
{
\"name\":\"Charlie Green\"
},
{
\"name\":\"Alex Adams\"
},
{
\"name\":\"Morgan Thompson\"
},
{
\"name\":\"Taylor Clark\"
},
{
\"name\":\"Jordan Hernandez\"
},
{
\"name\":\"Jamie Lewis\"
}
]
}
}
Then we send intermediary states – someone scores goal, gets yellow card, being subsctituted. And here is the end message in simulation (final scores, final events attached to corresponding players):
\n{
\"homeTeam\":{
\"name\":\"Real Madrid\",
\"score\":3,
\"players\":[
{
\"name\":\"John Doe\",
\"events\":[
{
\"type\":\"YELLOW_CARD\",
\"minute\":6
},
{
\"type\":\"SUBSTITUTE\",
\"minute\":39
}
]
},
{
\"name\":\"Jane Smith\"
},
{
\"name\":\"Alex Johnson\"
},
{
\"name\":\"Chris Lee\",
\"events\":[
{
\"type\":\"GOAL\",
\"minute\":84
}
]
},
{
\"name\":\"Pat Kim\"
},
{
\"name\":\"Sam Morgan\"
},
{
\"name\":\"Jamie Brown\",
\"events\":[
{
\"type\":\"SUBSTITUTE\",
\"minute\":9
}
]
},
{
\"name\":\"Casey Davis\",
\"events\":[
{
\"type\":\"YELLOW_CARD\",
\"minute\":81
}
]
},
{
\"name\":\"Morgan Garcia\",
\"events\":[
{
\"type\":\"SUBSTITUTE\",
\"minute\":15
},
{
\"type\":\"GOAL\",
\"minute\":30
},
{
\"type\":\"YELLOW_CARD\",
\"minute\":57
},
{
\"type\":\"GOAL\",
\"minute\":62
},
{
\"type\":\"RED_CARD\",
\"minute\":66
}
]
},
{
\"name\":\"Taylor White\",
\"events\":[
{
\"type\":\"YELLOW_CARD\",
\"minute\":18
},
{
\"type\":\"SUBSTITUTE\",
\"minute\":42
},
{
\"type\":\"SUBSTITUTE\",
\"minute\":45
},
{
\"type\":\"YELLOW_CARD\",
\"minute\":69
},
{
\"type\":\"RED_CARD\",
\"minute\":72
}
]
},
{
\"name\":\"Jordan Martinez\",
\"events\":[
{
\"type\":\"SUBSTITUTE\",
\"minute\":21
},
{
\"type\":\"SUBSTITUTE\",
\"minute\":24
}
]
}
]
},
\"awayTeam\":{
\"name\":\"Barcelona\",
\"score\":3,
\"players\":[
{
\"name\":\"Robin Wilson\"
},
{
\"name\":\"Drew Taylor\",
\"events\":[
{
\"type\":\"RED_CARD\"
},
{
\"type\":\"GOAL\",
\"minute\":12
}
]
},
{
\"name\":\"Jessie Bailey\"
},
{
\"name\":\"Casey Flores\",
\"events\":[
{
\"type\":\"YELLOW_CARD\",
\"minute\":78
}
]
},
{
\"name\":\"Jordan Walker\",
\"events\":[
{
\"type\":\"SUBSTITUTE\",
\"minute\":33
}
]
},
{
\"name\":\"Charlie Green\",
\"events\":[
{
\"type\":\"GOAL\",
\"minute\":51
},
{
\"type\":\"GOAL\",
\"minute\":60
},
{
\"type\":\"SUBSTITUTE\",
\"minute\":75
}
]
},
{
\"name\":\"Alex Adams\"
},
{
\"name\":\"Morgan Thompson\",
\"events\":[
{
\"type\":\"YELLOW_CARD\",
\"minute\":27
},
{
\"type\":\"SUBSTITUTE\",
\"minute\":48
}
]
},
{
\"name\":\"Taylor Clark\",
\"events\":[
{
\"type\":\"SUBSTITUTE\",
\"minute\":3
},
{
\"type\":\"SUBSTITUTE\",
\"minute\":87
}
]
},
{
\"name\":\"Jordan Hernandez\"
},
{
\"name\":\"Jamie Lewis\",
\"events\":[
{
\"type\":\"YELLOW_CARD\",
\"minute\":36
},
{
\"type\":\"SUBSTITUTE\",
\"minute\":54
}
]
}
]
}
}
When we used Protobuf encoding for game state we serialized the data according to this Protobuf schema:
\nsyntax = \"proto3\";
package centrifugal.centrifuge.examples.compression_playground;
option go_package = \"./;apppb\";
enum EventType {
UNKNOWN = 0; // Default value, should not be used
GOAL = 1;
YELLOW_CARD = 2;
RED_CARD = 3;
SUBSTITUTE = 4;
}
message Event {
EventType type = 1;
int32 minute = 2;
}
message Player {
string name = 1;
repeated Event events = 2;
}
message Team {
string name = 1;
int32 score = 2;
repeated Player players = 3;
}
message Match {
int32 id = 1;
Team home_team = 2;
Team away_team = 3;
}
Below are the results of our experiment, comparing different protocols and compression settings:
\n| Protocol | Compression | Delta | Bytes sent | Percentage |
|---|---|---|---|---|
| JSON over JSON | No | No | 40251 | 100.0 (base) |
| JSON over JSON | Yes | No | 15669 | 38.93 |
| JSON over JSON | No | Yes | 6043 | 15.01 |
| JSON over JSON | Yes | Yes | 5360 | 13.32 |
| -- | -- | -- | -- | -- |
| JSON over Protobuf | No | No | 39180 | 97.34 |
| JSON over Protobuf | Yes | No | 15542 | 38.61 |
| JSON over Protobuf | No | Yes | 4287 | 10.65 |
| JSON over Protobuf | Yes | Yes | 4126 | 10.25 |
| -- | -- | -- | -- | -- |
| Protobuf over Protobuf | No | No | 16562 | 41.15 |
| Protobuf over Protobuf | Yes | No | 13115 | 32.58 |
| Protobuf over Protobuf | No | Yes | 4382 | 10.89 |
| Protobuf over Protobuf | Yes | Yes | 4473 | 11.11 |
Let's now discuss the results we observed in detail.
\nIn this case we are sending JSON data with football match game state over JSON Centrifugal protocol.
\nJSON over JSON (No Compression, No Delta)\nBytes Sent: 40251\nPercentage: 100.0%\nAnalysis: This is a baseline scenario, with no compression and no delta, results in the highest amount of data being sent. But very straightforward to implement.
\nJSON over JSON (With Compression, No Delta)\nBytes Sent: 15669\nPercentage: 38.93%\nAnalysis: Enabling compression reduces the data size significantly to 38.93% of the original, showcasing the effectiveness of deflate compression. See how to configure compression in Centrifugo, note that it comes with CPU and memory overhead which depends on your load profile.
\nJSON over JSON (No Compression, With Delta)\nBytes Sent: 6043\nPercentage: 15.01%\nAnalysis: Using delta compression without deflate compression reduces data size to 15.01% for this use case, only changes are being sent after the initial full payload. See how to enable delta compression in channels in Centrifugo. The nice thing about using delta compression instead of deflate compression is that deltas require less and more predictable resource overhead.
\nJSON over JSON (With Compression and Delta)\nBytes Sent: 5360\nPercentage: 13.32%\nAnalysis: Combining both compression and delta further reduces the data size to 13.32%, achieving the highest efficiency in this category. The benefit is not huge, because we already send small deltas here.
\nIn this case we are sending JSON data with football match game state over Protobuf Centrifugal protocol.
\nJSON over Protobuf (No Compression, No Delta)\nBytes Sent: 39180\nPercentage: 97.34%\nAnalysis: Switching to Protobuf encoding of Centrifugo protocol but still sending JSON data slightly reduces the data size to 97.34% of the JSON over JSON baseline. The benefit here comes from the fact Centrifugo does not introduce a lot of its own protocol overhead – Protobuf is more compact. But we still send JSON data as Protobuf payloads – that's why it's generally comparable with a baseline.
\nJSON over Protobuf (With Compression, No Delta)\nBytes Sent: 15542\nPercentage: 38.61%\nAnalysis: Compression with Protobuf encoding brings similar benefits as with JSON, reducing the data size to 38.61%.
\nJSON over Protobuf (No Compression, With Delta)\nBytes Sent: 4287\nPercentage: 10.65%\nAnalysis: Delta compression with Protobuf is effective, reducing data to 10.65%. It's almost x10 reduction in bandwidth compared to the baseline!
\nI guess at this point you may be curious how delta frames look like in case of JSON protocol. Here is a screenshot:
\n![\"delta](\"https://centrifugal.dev/assets/images/delta_frames-7d915a6b62f3cbcbfa4e0a1d738e79df.png\")
In this case we are sending Protobuf binary data with football match game state over Protobuf Centrifugal protocol.
\nProtobuf over Protobuf (No Compression, No Delta)\nBytes Sent: 16562\nPercentage: 41.15%\nAnalysis: Using Protobuf for both encoding and transmission without any compression or delta reduces data size to 41.15%. So you may get the most efficient setup with nice bandwidth reduction. But the cost is more complex data encoding.
\nProtobuf over Protobuf (With Compression, No Delta)\nBytes Sent: 13115\nPercentage: 32.58%\nAnalysis: Compression reduces the data size to 32.58%. Note, that in this case it's not very different from JSON case.
\nProtobuf over Protobuf (No Compression, With Delta)\nBytes Sent: 4382\nPercentage: 10.89%\nAnalysis: Delta compression is again very effective here, reducing the data size to 10.89%. Again - comparable to JSON case.
\nProtobuf over Protobuf (With Compression and Delta)\nBytes Sent: 4473\nPercentage: 11.11%\nAnalysis: Combining both methods results in a data size of 11.11%. Even more than in JSON case. That's bacause binary data is not compressed very well with deflate algorithm.
\nWebSocket permessage-deflate compression significantly reduces the amount of data transferred over WebSocket connections. While it incurs CPU and memory overhead, it may be still worth using from a total cost perspective.
\nDelta compression makes perfect sense for channels where data changes only slightly between publications. In our experiment, it resulted in a tenfold reduction in bandwidth usage.
\nUsing binary data in combination with the Centrifugo Protobuf protocol provides substantial bandwidth reduction even without deflate or delta compression. However, this comes at the cost of increased data format complexity. An additional benefit of using the Centrifugo Protobuf protocol is its faster marshalling and unmarshalling on the server side compared to the JSON protocol.
\nFor Centrifugo, these results highlighted the potential of implementing delta compression, so we proceeded with it. The benefit depends on the nature of the data being sent – you can achieve even greater savings if you have larger messages that are very similar to each other.
", "url": "https://centrifugal.dev/blog/2024/05/30/real-time-data-compression-experiments", "title": "Experimenting with real-time data compression by simulating a football match events", "summary": "This post shows the potential profit of enabling delta compression in channels and demonstrates the reduction of data transfer in various scenarios, including different Centrifugo protocol formats and using WebSocket permessage-deflate compression.", "date_modified": "2024-05-30T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "websocket", "compression" ] }, { "id": "https://centrifugal.dev/blog/2024/03/18/stream-loki-logs-to-browser-with-websocket-to-grpc-subscriptions", "content_html": "As of version 5.1.0, Centrifugo introduces an experimental yet powerful extension that promises to simplify the data delivery process to the browser using GRPC streams. We believe it may help you to solve some practical tasks in minutes. Let's dive into how this feature works and how you can leverage it in your applications integrating with Loki real-time log streaming capabilities.
\nProxy Subscription Streams support pushing data directly to Centrifugo client channel subscriptions from your application backend over GRPC streams. This feature is designed to facilitate individual data streams to clients as soon as they subscribe to a channel, acting as a bridge between WebSocket connections from clients and GRPC streams to the backend. It supports both unidirectional (backend to client) and bidirectional (both ways) streams, thereby enhancing flexibility in data streaming.
\n
The design is inspired by Websocketd server – but while Websocketd transforms data from programs running locally, Centrifugo provides a more generic network interface with GRPC. And all other features of Centrifugo like connection authentication, online presence come as a great bonus.
\nIn the documentation for Proxy Subscription Streams we mentioned streaming logs from Loki as one of the possible use cases. Let's expand on the idea and implement the working solution in just 10 minutes.
\nHere is a demo of what we well get:
\n\nTake a look at full source code on Github.
\nLoki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate, making it a perfect candidate for our real-time log streaming example.
\nWe will build the example using Docker Compose, all we have to do for the example is to include Loki image to docker-compose.yml:
services:
loki:
image: grafana/loki:2.9.5
ports:
- \"3100:3100\"
Loki can ingest logs via various methods, including Promtail, Grafana Agent, Fluentd, and more. For simplicity, we will send logs to Loki ourselves from the Go application.
\nTo send logs to Loki, we can use the HTTP API that Loki provides. This is a straightforward way to push logs directly from an application. The example below demonstrates how to create a simple Go application that generates logs and sends them to Loki using HTTP POST requests.
\nFor this post we will be using Go language to implement the backend part. But it could be any other programming language.
\nFirst, let's some code to send a log entries to Loki:
\nconst (
\tlokiPushEndpoint = \"http://loki:3100/loki/api/v1/push\"
)
type lokiPushMessage struct {
\tStreams []lokiStream `json:\"streams\"`
}
type lokiStream struct {
\tStream map[string]string `json:\"stream\"`
\tValues [][]string `json:\"values\"`
}
func sendLogMessageToLoki(_ context.Context) error {
\tsources := []string{\"backend1\", \"backend2\", \"backend3\"}
\tsource := sources[rand.Intn(len(sources))]
\tlogMessage := fmt.Sprintf(\"log from %s source\", source)
\tpayload := lokiPushMessage{
\t\tStreams: []lokiStream{
\t\t\t{
\t\t\t\tStream: map[string]string{
\t\t\t\t\t\"source\": source,
\t\t\t\t},
\t\t\t\tValues: [][]string{
\t\t\t\t\t{fmt.Sprintf(\"%d\", time.Now().UnixNano()), logMessage},
\t\t\t\t},
\t\t\t},
\t\t},
\t}
\tjsonData, err := json.Marshal(payload)
\tif err != nil {
\t\treturn err
\t}
\tresp, err := http.Post(
\t\tlokiPushEndpoint, \"application/json\", bytes.NewBuffer(jsonData))
\tif err != nil {
\t\treturn err
\t}
\tdefer resp.Body.Close()
\tif resp.StatusCode != http.StatusNoContent {
\t\treturn fmt.Errorf(\"unexpected status code: %d\", resp.StatusCode)
\t}
\treturn nil
}
func sendLogsToLoki(ctx context.Context) {
\tfor {
\t\tselect {
\t\tcase <-ctx.Done():
\t\t\treturn
\t\tcase <-time.After(200 * time.Millisecond):
\t\t\terr := sendLogMessageToLoki(ctx)
\t\t\tif err != nil {
\t\t\t\tlog.Println(\"error sending log to Loki:\", err)
\t\t\t\tcontinue
\t\t\t}
\t\t}
\t}
}
func main() {
\tctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
\tdefer cancel()
\tsendLogsToLoki(ctx)
}
This program defines a sendLogsToLoki function that constructs a log entry and sends it to Loki using its HTTP API. It continuously generates log messages every 200 milliseconds.
The lokiPushMessage struct is structured to match the JSON payload expected by Loki's /loki/api/v1/push endpoint. Each log entry consists of a set of labels (in the Stream map) and log line values, where each value is a two-element array containing the timestamp and the log line. The timestamp is in nanoseconds to match Loki's expected format.
Note, in the example we randomly set log entry source label choosing between backend1, backend2 and backend3 values.
At this point our program pushes some logs to Loki, now let's add Centrifugo to consume them from browser in real-time.
\nAdding Centrifugo is also rather straightforward:
\nservices:
centrifugo:
image: centrifugo/centrifugo:v5.3.0
restart: unless-stopped
volumes:
- ./centrifugo/config.json:/centrifugo/config.json
command: centrifugo -c config.json
expose:
- 8000
Where config.json is:
{
\"client_insecure\": true,
\"allowed_origins\": [\"http://localhost:9000\"],
\"proxy_subscribe_stream_endpoint\": \"grpc://backend:12000\",
\"proxy_subscribe_stream_timeout\": \"3s\",
\"namespaces\": [
{
\"name\": \"logs\",
\"proxy_subscribe_stream\": true
}
]
}
Note, we enabled client_insecure option here – this is to keep example short, but in real live you may benefit from Centrifugo authentication: JWT-based or proxy-based.
<!DOCTYPE html>
<html lang=\"en\">
<head>
<meta charset=\"UTF-8\">
<title>Streaming logs with Centrifugo and Loki</title>
</head>
<body>
<div id=\"app\">
<form id=\"input\" onsubmit=\"subscribeToLogs(event)\">
<input type=\"text\" id=\"query\" autocomplete=\"off\" placeholder=\"Enter log query\" />
<button id=\"submit\" type=\"submit\">SUBSCRIBE</button>
</form>
<div id=\"logs\" style=\"margin-top: 20px;\">
<ul id=\"lines\"></ul>
</div>
</div>
<script src=\"https://unpkg.com/centrifuge@^5/dist/centrifuge.js\"></script>
<script src=\"app.js\"></script>
</body>
</html>
In the final version we've also included some CSS to this HTML to make it look a bit nicer.
\nAnd our Javascript code in app.js:
const logs = document.getElementById('logs');
const lines = document.getElementById('lines');
const queryInput = document.getElementById('query');
const button = document.getElementById('submit');
function subscribeToLogs(e) {
e.preventDefault();
const query = queryInput.value;
if (!query) {
alert('Please enter a query.');
return;
}
queryInput.disabled = true;
button.disabled = true;
const centrifuge = new Centrifuge('ws://localhost:9000/connection/websocket');
const subscription = centrifuge.newSubscription('logs:stream', {
data: { query: query }
});
subscription.on('publication', function(ctx) {
const logLine = ctx.data.line;
const logItem = document.createElement('li');
logItem.textContent = logLine;
lines.appendChild(logItem);
logs.scrollTop = logs.scrollHeight;
});
subscription.subscribe();
centrifuge.connect();
}
In the final example we've also added Nginx container to serve static files and proxy WebSocket connections to Centrifugo. Check it out in the source code.
\nWhen user enters Loki query to input, subscription goes to Centrifugo and Centrifugo then realizes it's a proxy stream subscription (since channel belongs to logs channel namespace). Centrifugo then calls the backend GRPC endpoint (backend:12000) and expect it to implement unidirectional GRPC stream contract. Our last part here - to implement it.
On your backend, we'll implement a GRPC service that interacts with Loki to tail logs and then re-send them to Centrifugo subscription stream. Let's implement such service.
\nWe first need to take Centrifugo proxy.proto definitions. And we will implement SubscribeUnidirectional method from it.
You need to install protoc, also install plugins for Go and GRPC:
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
And then:
\nprotoc -I ./ proxy.proto --go_out=./ --go-grpc_out=./
This will generate Protobuf messages and GRPC code required for writing GRPC service. We can use generated definitions now:
\nimport (
\t\"log\"
\t\"fmt\"
\tpb \"backend/internal/proxyproto\"
\t\"github.com/grafana/loki/pkg/logproto\"
\t\"google.golang.org/grpc\"
\t\"google.golang.org/grpc/credentials/insecure\"
)
const (
\tlokiGRPCAddress = \"loki:9095\"
)
type streamerServer struct {
\tpb.UnimplementedCentrifugoProxyServer
\tlokiQuerierClient logproto.QuerierClient
}
type clientData struct {
\tQuery string `json:\"query\"`
}
func (s *streamerServer) SubscribeUnidirectional(
\treq *pb.SubscribeRequest,
\tstream pb.CentrifugoProxy_SubscribeUnidirectionalServer,
) error {
\tvar cd clientData
\terr := json.Unmarshal(req.Data, &cd)
\tif err != nil {
\t\treturn fmt.Errorf(\"error unmarshaling data: %w\", err)
\t}
\tquery := &logproto.TailRequest{
\t\tQuery: cd.Query,
\t}
\tctx, cancel := context.WithCancel(stream.Context())
\tdefer cancel()
\tlogStream, err := s.lokiQuerierClient.Tail(ctx, query)
\tif err != nil {
\t\treturn fmt.Errorf(\"error querying Loki: %w\", err)
\t}
\tstarted := time.Now()
\tlog.Println(\"unidirectional subscribe called with request\", req)
\tdefer func() {
\t\tlog.Println(\"unidirectional subscribe finished, elapsed\", time.Since(started))
\t}()
\terr = stream.Send(&pb.StreamSubscribeResponse{
\t\tSubscribeResponse: &pb.SubscribeResponse{},
\t})
\tif err != nil {
\t\treturn err
\t}
\tfor {
\t\tselect {
\t\tcase <-stream.Context().Done():
\t\t\treturn stream.Context().Err()
\t\tdefault:
\t\t\tresp, err := logStream.Recv()
\t\t\tif err != nil {
\t\t\t\treturn fmt.Errorf(\"error receiving from Loki stream: %v\", err)
\t\t\t}
\t\t\tfor _, entry := range resp.Stream.Entries {
\t\t\t\tline := fmt.Sprintf(\"%s: %s\", entry.Timestamp.Format(\"2006-01-02T15:04:05.000Z07:00\"), entry.Line)
\t\t\t\terr = stream.Send(&pb.StreamSubscribeResponse{
\t\t\t\t\tPublication: &pb.Publication{Data: []byte(`{\"line\":\"` + line + `\"}`)},
\t\t\t\t})
\t\t\t\tif err != nil {
\t\t\t\t\treturn err
\t\t\t\t}
\t\t\t}
\t\t}
\t}
}
func main() {
\tquerierConn, err := grpc.Dial(lokiGRPCAddress, grpc.WithTransportCredentials(insecure.NewCredentials()))
\tif err != nil {
\t\tlog.Fatalf(\"failed to dial Loki: %v\", err)
\t}
\tquerierClient := logproto.NewQuerierClient(querierConn)
\taddr := \":12000\"
\tlis, err := net.Listen(\"tcp\", addr)
\tif err != nil {
\t\tlog.Fatalf(\"failed to listen: %v\", err)
\t}
\ts := grpc.NewServer(grpc.MaxConcurrentStreams(math.MaxUint32))
\tpb.RegisterCentrifugoProxyServer(s, &streamerServer{
\t\tlokiQuerierClient: querierClient,
\t})
\tlog.Println(\"Server listening on\", addr)
\tif err := s.Serve(lis); err != nil {
\t\tlog.Fatalf(\"failed to serve: %v\", err)
\t}
}
Things to note:
\n/loki/api/v1/tail but this would mean establishing new connection for every tail operation - with GRPC we can use many concurrent tail requests all multiplexed over a single network connection.Note, we bypass some security considerations in this example. In practice you must be more careful with query supplied by user in the form - validate and sanitize it before passing to Loki. Proxy subscription GRPC contract allows you to communicate custom errors with the client-side.
Subscription streams may be a very powerful generic feature in your arsenal. Here we've shown how simple it could be to make a proof of concept of the real-time application which consumes individual data from third-party streaming provider.
\nCentrifugo provides WebSocket SDKs for popular languages used to build UI layer, provides authentication and proper management of real-time connections. And with subscription streams feature Centrifugo gives you an answer on how to quickly translate real-time data based on individual query to user.
", "url": "https://centrifugal.dev/blog/2024/03/18/stream-loki-logs-to-browser-with-websocket-to-grpc-subscriptions", "title": "Stream logs from Loki to browser with Centrifugo Websocket-to-GRPC subscriptions", "summary": "Centrifugo has GRPC subscription streams feature, in this post we show how this feature may simplify a task of delivering data to application UI in real-time. We integrate with Loki, injest log entries and stream logs to the browser based on user-supplied query", "date_modified": "2024-03-18T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "loki", "grpc" ] }, { "id": "https://centrifugal.dev/blog/2023/10/29/discovering-centrifugo-pro-push-notifications", "content_html": "In our v5 release post, we announced the upcoming launch of Centrifugo PRO. We are happy to say that it was released soon after that, and at this point, we already have several customers of the PRO version.
\nI think it's time to look at the current state of the PRO version and finally start talking more about its benefits. In this post, we will talk more about one of the coolest PRO features we have at this point: the push notifications API.
\nWhen Centrifugo was originally created, its main goal was to help introduce real-time messaging features to existing systems, written in traditional frameworks which work on top of the worker/thread model. Serving many concurrent connections is a non-trivial task in general, and without native efficient concurrency support, it becomes mostly impossible without a shift in the technology stack. Integrating with Centrifugo makes it simple to introduce an efficient real-time layer, while keeping the existing application architecture.
\nAs time went on, Centrifugo got some unique features which now justify its usage even in conjunction with languages/frameworks with good concurrency support. Simply using Centrifugo for at-most-once PUB/SUB may already save a lot of development time. The task, which seems trivial at first glance, has a lot of challenges in practice: client SDKs with reconnect and channel multiplexing, scalability to many nodes, WebSocket fallbacks, etc.
\nThe combination of useful possibilities has made Centrifugo an attractive component for building enterprise-level applications. Let's be honest here - for pet projects, developers often prefer writing WebSocket communications themselves, and Centrifugo may be too heavy and an extra dependency. But in a corporate environment, the decision on which technology to use should take into account a lot of factors, like those we just mentioned above. Using a mature technology is often preferred to building from scratch and making all the mistakes along the way.
\nWith the PRO version, our goal is to provide even more value for established businesses when switching to Centrifugo. We want to solve tricky cases and simplify them for our customers; we want to step into related areas where we see we can provide sufficient value.
\nOne rule we try to follow for PRO features that extend Centrifugo’s scope is this: we are not trying to replicate something that already exists in other systems, but rather, we strive to improve upon it. We focus on solving practical issues that we observe, providing a unique value proposition for our customers. This post describes one such example — we will demonstrate our approach to push notifications, which is one the features of Centrifugo PRO.
\n![](\"https://centrifugal.dev/img/push_characters.jpg\")
\nWhy provide a push notifications API at all? Well, actually, real-time messages and push notifications are so close that many developers hardly see the difference before starting to work with both more closely.
\nI’ve heard several stories where chat functionality on mobile devices was implemented using only native push notifications — without using a separate real-time transport like WebSocket while the app is in the foreground. While this is not a recommended approach due to the delivery properties of push notifications, it proves that real-time messages and push notifications are closely related concepts and sometimes may interchange with each other.
\nWhen developers introduce WebSocket communication in an application, they often ask the question—what should I do next to deliver some important messages to a user who is currently not actively using the application? WebSockets are great when the app is in the foreground, but when the app goes to the background, the recommended approach is to close the WebSocket connection. This is important to save battery, and operating systems force the closing of connections after some time anyway.
\nThe delivery of important app data is then possible over push notifications. See a good overview of them on web.dev.
\nPreviously, Centrifugo positioned itself solely as a transport layer for real-time messages. In our FAQ, we emphasized this fact and suggested using separate software products to send push notifications.
\nNow, with Centrifugo PRO, we provide this functionality to our customers. We have extended our server API with methods to manage and send push notifications. I promised to tell you why we believe our implementation is super cool. Let’s dive into the details.
\nPush notifications are super handy, but there’s a bit to do to get them working right. Let's break it down!
\nWe believe that we were able to achieve a unique combination of design decisions which allows us to provide push notification support like no one else provides. Let’s dive into what makes our approach special!
\nWhen providing the push notification feature, other solutions like Pusher or Ably also offer their own SDKs for managing notifications on the client side.
\nWhat we've learned, though, during the Centrifugo life cycle, is that creating and maintaining client SDKs for various environments (iOS, Android, Web, Flutter) is one of the hardest parts of the Centrifugo project.
\nSo the decision here was simple and natural: Centrifugo PRO does not introduce any client SDKs for push notifications on the client side.
\nWhen integrating with Centrifugo, you can simply use the native SDKs provided by each platform. We bypass the complexities of SDK development and concentrate on server-side improvements. With this decision, we are not introducing any limitations to the client side.
\nYou get:
\nThis approach was not possible with our real-time SDKs, as WebSocket communication is very low-level, and Centrifugo’s main goal was to provide some high-level features on top of it. However, with push notifications, proceeding without a custom SDK seems like a choice beneficial for everyone.
\nThe main work we did was on the server side. Let's go through the entire workflow of push notification delivery and describe what Centrifugo PRO provides for each step.
\nLet's suppose you got the permission from the user and received the device push token. At this point you must save it to database for sending notifications later using this token. Centrifugo PRO provides API called device_register to do exactly this.
\nAt this point, we use PostgreSQL for storing tokens – which is a very popular SQL database. Probably we will add more storage backend options in the future.
\nWhen calling Centrifugo device_register API you can provide user ID, list of topics to subscribe, platform from which the user came from (ios, android, web), also push notifications provider. To deliver push notifications to devices Centrifugo PRO integrates with the following push notification providers:
![\"Push\"](\"https://centrifugal.dev/assets/images/push_notifications-97d3e940efd8a02ab6781f048031f100.png\")
So we basically cover all the most popular platforms out of the box.
\nAfter registering the device token, Centrifugo PRO returns a device_id to you. This device ID must be stored on the client device. As long as the frontend has this device_id, it can update the device's push token information from time to time to keep it current (by just calling device_register again, but with device_id attached).
After saving the token, your backend can start sending push notifications to devices.
\nTo send push notifications we provide another API called send_push_notification. You need to provide some filter in the API request to tell Centrifugo who you want to send notification. You also need to provide push notification payload. For example, using Centrifugo HTTP API:
\ncurl -X POST http://localhost:8000/api/send_push_notification \\
-H \"Authorization: apikey <KEY>\" \\
-d @- <<'EOF'
{
\"recipient\": {
\"filter\": {
\"topics\": [\"test\"]
}
},
\"notification\": {
\"fcm\": {
\"message\": {
\"notification\": {\"title\": \"Hello\", \"body\": \"How are you?\"}
}
}
}
}
EOF
Here is another important decision we made: Centrifugo PRO allows you to specify raw JSON objects for each provider we support. In other words, we do not wrap the push notifications API for FCM, APNS, HMS - we give you a way to construct the entire push notification message.
\nThis means the Centrifugo push API supports all the fields of push notification payloads out-of-the-box, for all push providers. You can simply use the documentation of FCM, APNs, and send the constructed requests to Centrifugo. There is no need for us to update Centrifugo PRO in any way to support new fields added by providers to push APIs.
\nWhen you send a push notification with a filter and push payload for each provider you want, it's queued by Centrifugo. We use Redis Streams for queuing and optionally a queue based on PostgreSQL (less efficient, but still robust enough).
\nThe fact that the notification is being queued means a very fast response time – so you can integrate with Centrifugo from within the hot paths of your application backend. You may additionally provide a push expiration time and a unique push identifier. If you have not provided a unique identifier, Centrifugo generates one for you and returns it in the response. The unique identifier may later be used to track push status in Centrifugo PRO's push notification analytics.
\nWe then have efficient workers which process the queue with minimal latency and send push notifications using batch requests for each provider - i.e., we do this in the most effective way possible. We conducted a benchmark of our worker system with FCM – and we can easily send several million pushes per minute.
\nAnother decision we made - Centrifugo PRO supports sending push notifications to a raw list of tokens. This makes it possible for our customers to use their own token storage. For example, such storage could already exist before you started using Centrifugo, or you might need a different storage/schema. In such cases, you can use Centrifugo just as an effective push sender server.
\nFinally, Centrifugo PRO supports sending delayed push notification - to queue push for a later delivery, so for example you can send notification based on user time zone and let Centrifugo PRO send it when needed. Or you may send slightly delayed push notification together with real-time message and if client provided an ack to real-time message - cancel push notification.
\nFCM and HMS have a built-in way of sending notification to large groups of devices over topics mechanism (the same for HMS). One problem with native FCM or HMS topics though is that device can subscribe to any topic from the frontend side without any permission check. In today's world this is usually not desired. So Centrifugo PRO re-implements FCM and HMS topics by introducing an additional API to manage device subscriptions to topics.
\nCentrifugo PRO device topic subscriptions also add a way to introduce the missing topic semantics for APNs.
\nCentrifugo PRO additionally provides an API to create persistent bindings of user to notification topics. See user_topic_list and user_topic_update. As soon as user registers a device – it will be automatically subscribed to its own topics pre-created over the API. As soon as user logs out from the app and you update user ID of the device - user topics binded to the device automatically removed/switched.
\nThis design solves one of the issues with push notifications (with FCM in particular) – if two different users use the same device it's becoming problematic to unsubscribe the device from large number of topics upon logout. Also, as soon as user to topic binding added (using user_topic_update API) – it will be synchronized across all user active devices. You can still manage such persistent subscriptions on the application backend side if you prefer and provide the full list inside device_register call - Centrifugo PRO API gives you freedom here.
Centrifugo PRO offers the ability to inspect sent push notifications using ClickHouse analytics. Push providers may also offer their own analytics, such as FCM, which provides insight into push notification delivery. Centrifugo PRO also offers a way to analyze push notification delivery and interaction using the update_push_status API. This API allows updating ClickHouse table and add status for each push sent:
\ndeliveredinteractedIt's then possible to make queries to ClickHouse and build various analytical reports. Or use ClickHouse for real-time graphs - for example, from Grafana.
\nFinally, Centrifugo PRO provides a simple web UI for inspecting registered devices. It can simplify development, provide a way to look at live data, and send simple push notification alerts to users or topics.
\n![](\"https://centrifugal.dev/assets/images/push_ui-01989161306e71d882a064b605395dcb.png\")
We really believe in our push notifications and will be working hard to make them even better. The API we already have serves well to cover common push notification delivery use cases, but we won't stop here. Some areas for improvements are: functionality of built-in push notifications web UI, extending push analytics by providing user friendly UI for the insights about push delivery and engagement. The good thing is that we already have a ground for making this.
\nTake a look at the documentation of Centrifugo PRO push notification API for more formal details and some things not mentioned here. Probably at the time you are reading this we already added something great to the API.
\nEven though Centrifugo PRO is pretty new, it already has a lot of helpful features, and we have plans to add even more. You can see what’s coming up next on our Centrifugo PRO planned features board. We're excited to share more blog posts like this one in the future.
", "url": "https://centrifugal.dev/blog/2023/10/29/discovering-centrifugo-pro-push-notifications", "title": "Discovering Centrifugo PRO: push notifications API", "summary": "We start talking more about recently launched Centrifugo PRO. In this post, we share details about Centrifugo PRO push notification API implementation - how it works and what makes it special and practical.", "date_modified": "2023-10-29T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "pro", "push notifications" ] }, { "id": "https://centrifugal.dev/blog/2023/08/29/using-centrifugo-in-rabbitx", "content_html": "This post introduces a new format in Centrifugal blog – interview with a Centrifugo user! Let's dive into an exciting chat with the engineering team of RabbitX platform, a global permissionless perpetuals exchange powered on Starknet. We will discover how Centrifugo helped RabbitX to build a broker platform with current trading volume of 25 million USD daily! 🚀🎉
\nRabbitX is a global permissionless perpetuals exchange built on Starknet. RabbitX is building the most secure and liquid global derivatives network, giving you 24/7 access to global markets anywhere in the world, with 20x leverage. In its core there is an orderbook - where traders match against market makers, which require to support high throughput and low latency tech stack.
\nThe technologies that we are using:
\nAlmost all the information users see in our terminal is streamed over Centrifugo. We use it for financial order books, candlestick chart updates, and stat number updates. We can also send real-time personal user notifications via Centrifugo. Instead of all the words, here is a short recording of our terminal trading BTC:
\n\nWell, that's an interesting thing. We heavily use Tarantool in our system. It grants us immense flexibility, performance, and the power to craft whatever we envision. It ensures the atomicity essential for trading match-making.
\nWhen we were in search of a WebSocket real-time bus for messages, we were pleasantly surprised to discover that Centrifugo integrates with Tarantool. In our scenario, this allowed us to bypass additional network round-trips, as we can stream data directly from Tarantool to Centrifugo channels. Reducing latency is paramount for financial instruments.
\nFurthermore, I can mention that over our nine months in production, we didn't encounter any issues with Centrifugo – it performed flawlessly!
\nRegarding authentication, we employ Centrifugo's JWT authentication and subscribe proxy. Thus, subscriptions are authorized on our specialized service written in Go. We're also actively using Centrifugo possibility to send initial channel data in the subscribe proxy response.
\nOne challenge we overcame was bridging the gap between the subscription's initial request and the continuous message stream in the order book component. To address this, we employed our own sequence numbers in events, coupled with Centrifugo's channel history – this allowed us to deal with missed events when needed. Actually the gaps in event stream are rare in practice and our workaround not needed most of the time, but now we're confident our users never experience this issue.
\nOur frontend is built on top of React in combination with TradingView Supercharts. And of course we are using centrifuge-js SDK for establishing connections with Centrifugo.
At this point we can have up to a thousand active concurrent traders and send more than 60 messages per second towards one client in peak hours. All the load is served with a single Centrifugo instance (and we have one standby instance).
\nWhen we designed the system the main goal was to have a homogeneous tech zoo, with a small amount of different technologies, to keep the number of failure points as small as possible. Tarantool is a sort of technology that really allows us to achieve this, we were able to add different decentralized mechanics to our system because of that. It’s not only an in-memory database, but in reality the app server as well.
\nIn our case, the fact Centrifugo supports Tarantool broker was a big discovery – the integration went smoothly, and everything has been working great since then.
", "url": "https://centrifugal.dev/blog/2023/08/29/using-centrifugo-in-rabbitx", "title": "Using Centrifugo in RabbitX", "summary": "In this post, the engineering team of RabbitX platform shares details about the usage of Centrifugo in their product.", "date_modified": "2023-08-29T00:00:00.000Z", "author": { "name": "Centrifugal Labs + RabbitX" }, "tags": [ "centrifugo", "interview", "usecase" ] }, { "id": "https://centrifugal.dev/blog/2023/08/19/asynchronous-message-streaming-to-centrifugo-with-benthos", "content_html": "Centrifugo provides HTTP and GRPC APIs for publishing messages into channels. Publish server API is very straighforward to use - it's a simple request with a channel and data to be delivered to active WebSocket connections subscribed to a channel.
\nSometimes though Centrifugo users want to avoid synchronous calls to Centrifugo API delegating this work to asynchronous tasks. Many companies have convenient infrastructure for messaging processing tasks - like Kafka, Nats Jetstream, Redis, RabbitMQ, etc. Some using transactional outbox pattern to reliably deliver events upon database changes and have a ready infrastructure to push such events to some queue. From which want to re-publish events to Centrifugo.
\nIn this post we get familiar with a tool called Benthos and show how it may simplify integrating your asynchronous message flow with Centrifugo. And we discuss some non-obvious pitfalls of asynchronous publishing approach in regards to real-time applications.
\nFirst start Centrifugo (with debug logging to see incoming API requests in logs):
\ncentrifugo genconfig
centrifugo -c config.json --log_level debug
Hope this step is already simple for you, if not - check out Quickstart tutorial.
\nBenthos is an awesome tool which allows consuming data from various inputs, process data, then output it into configured outputs. See more detailed description on Benthos' website.
\nThe number of inputs supported by Benthos is huge: check it out here. Most of the major systems widely used these days are supported. Benthos also supports many outputs – but here we only interested in message transfer to Centrifugo. There is no built-in Centrifugo output in Benthos but it provides a generic http_client output which may be used to send requests to any HTTP server. Benthos may also help with retries, provides tools for additional data processing and transformations.
![](\"https://centrifugal.dev/assets/images/benthos-1eb5dd899b464895e7b27fa75e0e6857.svg\")
Just like Centrifugo Benthos written in Go language – so its installation is very straighforward and similar to Centrifugo. See official instructions.
\nLet's assume you've installed Benthos and have benthos executable available in your system. Let's create Benthos configuration file:
benthos create > config.yaml
Take a look at generated config.yaml - it contains various options for Benthos instance, the most important (for the context of this post) are input and output sections.
And after that you can start Benthos instance with:
\nbenthos -c config.yaml
Now we need to tell Benthos from where to get data and how to send it to Centrifugo.
\nFor our example here we will user Redis List as an input, won't add any additional data processing and will output messages consumed from Redis List into Centrifugo publish server HTTP API.
\nTo achieve this add the following as input in Benthos config.yaml:
input:
label: \"centrifugo_redis_consumer\"
redis_list:
url: \"redis://127.0.0.1:6379\"
key: \"centrifugo.publish\"
And configure the output like this:
\noutput:
label: \"centrifugo_http_publisher\"
http_client:
url: \"http://localhost:8000/api/publish\"
verb: POST
headers:
X-API-Key: \"<CENTRIFUGO_API_KEY>\"
timeout: 5s
max_in_flight: 20
The output points to Centrifugo publish HTTP API. Replace <CENTRIFUGO_API_KEY> with your Centrifugo api_key (found in Centrifugo configuration file).
Start Benthos instance:
\nbenthos -c config.yaml
You will see errors while Benthos tries to connect to input Redis source. So start Redis server:
\ndocker run --rm -it --name redis redis:7
Now connect to Redis (using redis-cli):
docker exec -it redis redis-cli
And push command to Redis list:
\n127.0.0.1:6379> rpush centrifugo.publish '{\"channel\": \"chat\", \"data\": {\"input\": \"test\"}}'
(integer) 1
This message will be consumed from Redis list by Benthos and published to Centrifugo HTTP API. If you have active subscribers to channel chat – you will see messages delivered to them. That's it!
When using Redis List input you can scale out Benthos instances to run several of them if needed.
Here is a quick demonstration of the described integration. See how we push messages into Redis List and those are delivered to WebSocket clients:
\n\nThis all seems simple. But publishing messages asynchronously may highlight some pitfalls not visible or not applicable for synchronous publishing to Centrifugo API.
\nMost of the time it will work just fine. But one day you can come across intermediate queue growth and increased delivery lag. This may happen due to temporary Centrifugo or worker process unavailability. As soon as system comes back to normal queued messages will be delivered.
\nDepending on the real-time feature implemented this may be a concern to think about and decide whether this is desired or not. Your application should be designed accordingly to deal with late delivery.
\nBTW late delivery may be a case even with synchronous publishing – it just almost never strikes. But theoretically client can reload browser page and load initial app state while message flying from the backend to client over Centrifugo. It's not Centrifugo specific actually - it's just a nature of networks and involved latencies.
\nIn general solution to prevent late delivery UX issues completely is using object versioning. Object version should be updated in the database on every change from which the real-time event is generated. Attach object version information to every real-time message. Also get object version on initial state load. This way you can compare versions and drop non-actual real-time messages on client side.
\nPossible strategy may be using synchronous API for real-time features where at most once delivery is acceptable and use asynchronous delivery where you need to deliver messages with at least once guarantees. In a latter case you most probably designed proper idempotency behaviour on client side anyway.
\nAnother thing to consider is message ordering. Centrifugo itself may provide message ordering in channels. If you published one message to Centrifugo API, then another one – you can expect that messages will be delivered to a client in the same order. But as soon as you have an intermediary layer like Benthos or any other asynchronous worker process – then you must be careful about ordering. In case of Benthos and example here you can set max_in_flight parameter to 1 instead of 20 and keep only one instance of Benthos running to preserve ordering.
In case of streaming from Kafka you can rely on Kafka message partitioning feature to preserve message ordering.
\nIf you preserved ordering in your asynchronous workers then the next thing to consider is throughput limitations.
\nYou have a limited number of workers, these workers send requests to Centrifugo one by one. In this case throughput is limited by the number of workers and RTT (round-trip time) between worker process and Centrifugo.
\nIf we talk about using Redis List structure as a queue - you can possibly shard data amongst different Redis List queues by some key to improve throughput. In this case you need to push messages where order should be preserved into a specific queue. In this case your get a setup similar to Kafka partitioning.
\nIn case of using manually partitioned queues or using Kafka you can have parallelism equal to the number of partitions.
\nLet's say you have 20 workers which can publish messages in parallel and 5ms RTT time between worker and Centrifugo. In this case you can publish 20*(1000/5) = 4000 messages per second max.
\nTo improve throughput futher consider increasing worker number or batching publish requests to Centrifugo (using Centrifugo's batch API).
\nWhen publishing asynchronously you should also don't forget about error handling. Benthos will handle network errors automatically for you. But there could be internal errors from Centrifugo returned as part of response. It's not very convenient to handle with Benthos out of the box – so we think about adding transport-level error mode to Centrifugo.
\nSometimes you want to publish to Centrifugo asynchronously using messaging systems convenient for your company. Usually you can write worker process to re-publish messages to Centrifugo. Sometimes it may be simplified using helpful tools like Benthos.
\nHere we've shown how Benthos may be used to transfer messages from Redis List queue to Centrifugo API. With some modifications you can achieve the same for other input sources - such as Kafka, RabbitMQ, Nats Jetstream, etc.
\nBut publishing messages asynchronously highlights several pifalls - like late delivery, ordering issues, throughput considerations and error handling – which should be carefully addressed. Different real-time features may require different strategies.
", "url": "https://centrifugal.dev/blog/2023/08/19/asynchronous-message-streaming-to-centrifugo-with-benthos", "title": "Asynchronous message streaming to Centrifugo with Benthos", "summary": "In this post, we'll demonstrate how to asynchronously stream messages into Centrifugo channels from external data providers using Benthos tool. We also highlight some pitfalls which become more important in asynchronous publishing scenario.", "date_modified": "2023-08-19T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "benthos", "tutorial" ] }, { "id": "https://centrifugal.dev/blog/2023/06/29/centrifugo-v5-released", "content_html": "In Centrifugo v5 we're phasing out old client protocol support, introducing a more intuitive HTTP API, adjusting token management behaviour in SDKs, improving configuration process, and refactoring the history meta ttl option. As the result you get a cleaner, more user-friendly, and optimized Centrifugo experience. And we have important news about the project - check it out in the end of this post.
\nCentrifugo is an open-source scalable real-time messaging server. Centrifugo can instantly deliver messages to application online users connected over supported transports (WebSocket, HTTP-streaming, SSE/EventSource, GRPC, SockJS, WebTransport). Centrifugo has the concept of a channel – so it's a user-facing PUB/SUB server.
Centrifugo is language-agnostic and can be used to build chat apps, live comments, multiplayer games, real-time data visualizations, collaborative tools, etc. in combination with any backend. It is well suited for modern architectures and allows decoupling the business logic from the real-time transport layer.
Several official client SDKs for browser and mobile development wrap the bidirectional protocol. In addition, Centrifugo supports a unidirectional approach for simple use cases with no SDK dependency.
Let's proceed and take a look at most notable changes of Centrifugo v5.
\nWith the introduction of Centrifugo v4, our previous major release, we rolled out a new version of the client protocol along with a set of client SDKs designed to work in conjunction with it. Nevertheless, we maintained support for the old client protocol in Centrifugo v4 to facilitate a seamless migration of applications.
\nIn Centrifugo v5 we are discontinuing support for the old protocol. If you have been using Centrifugo v4 with the latest SDKs, this change should have no impact on you. From our perspective, removing support for the old protocol allows us to eliminate a considerable amount of non-obvious branching in the source code and cleanup Protobuf schema definitions.
\nIn Centrifugo v5 we are adjusting client SDK specification in the aspect of connection token management. Previously, returning an empty token string from getToken callback resulted in client disconnection with unauthorized reason.
There was some problem with it though. We did not take into account the fact that empty token may be a valid scenario actually. Centrifugo supports options to avoid using token at all for anonymous access. So the lack of possibility to switch between token/no token scenarios did not allow users to easily implement login/logout workflow. The only way was re-initializing SDK.
Now returning an empty string from getToken is a valid scenario which won't result into disconnect on the client side. It's still possible to disconnect client by returning a special error from getToken function.
And we are putting back setToken method to our SDKs – so it's now possible to reset the token to be empty upon user logout.
An abstract example in Javascript which demonstrates how login/logout flow may be now implemented with our SDK:
\nconst centrifuge = new Centrifuge('ws://localhost:8000/connection/websocket', {
// Provide function which returns empty string for anonymous users,
// and proper JWT for authenticated users.
getToken: getTokenImplementation
});
centrifuge.connect();
loginButton.addEventListener('click', function() {
centrifuge.disconnect();
// Do actual login here.
centrifuge.connect();
});
logoutButton.addEventListener('click', function() {
centrifuge.disconnect();
// Reset token - so that getToken will be called on next connect attempt.
centrifuge.setToken(\"\");
// Do actual logout here.
centrifuge.connect();
});
We updated all our SDKs to inherit described behaviour - check out v5 migration guide for more details.
\nOne of Centrifugo's key advantages for real-time messaging tasks is its ephemeral channels and per-channel history. In version 5, we've improved one aspect of handling history by offering users the ability to tune the history meta TTL option at the channel namespace level.
\nThe history meta TTL is the duration Centrifugo retains meta information about each channel stream, such as the latest offset and current stream epoch. This data allows users to successfully restore connections upon reconnection, particularly useful when subscribed to mostly inactive channels where publications are infrequent. Although the history meta TTL can usually be reasonably large (significantly larger than history TTL), there are certain scenarios where it's beneficial to minimize it as much as possible.
\nOne such use case is illustrated in this example. Using Centrifugo SDK and channels with history, it's possible to reliably stream results of asynchronous tasks to clients.
\nAs another example, consider a ChatGPT use case where clients ask questions, subscribe to a channel with the answer, and then the response is streamed towards the client token by token. This all may be done over a secure, separate channel protected with a token. With the ability to use a relatively small history meta TTL in the channel namespace, implementing such things is now simpler.
\nHence, history_meta_ttl is now an option at the channel namespace level (instead of per-engine). However, setting it is optional as we have a global default value for it - see details in the doc.
When running in cluster Centrifugo nodes can communicate between each other using broker's PUB/SUB. Nodes exchange some information - like statistics, emulation requests, etc.
\nIn Centrifugo v5 we are simplifying and making inter-node communication protocol slightly faster by removing extra encoding layers from it's format. Something similar to what we did for our client protocol in Centrifugo v4.
\nThis change, however, leads to an incompatibility between Centrifugo v4 and v5 nodes in terms of their communication protocols. Consequently, Centrifugo v5 cannot be part of a cluster with Centrifugo v4 nodes.
\nFrom the beginning Centrifugo HTTP API exposed one /api endpoint to make requests with all command types.
To work properly HTTP API had to add one additional layer to request JSON payload to be able to distinguish between different API methods:
\ncurl --header \"Content-Type: application/json\" \\
--header \"Authorization: apikey API_KEY\" \\
--request POST \\
--data '{\"method\": \"publish\", \"params\": {\"channel\": \"test\", \"data\": {\"x\": 1}}}' \\
http://localhost:8000/api
And it worked fine. It additionally supported request batching where users could send many commands to Centrifugo in one request using line-delimited JSON.
\nHowever, the fact that we were accommodating various commands via a single API endpoint resulted in nested serialized payloads for each command. The top-level method would determine the structure of the params. We addressed this issue in the client protocol in Centrifugo v4, and now we're addressing a similar issue in the inter-node communication protocol in Centrifugo v5.
\nAt some point we introduced GRPC API in Centrifugo. In GRPC case we don't have a way to send batches of commands without defining a separate method to do so.
\nThese developments highlighted the need for us to align the HTTP API format more closely with the GRPC API. Specifically, we need to separate the command method from the actual method payload, moving towards a structure like this:
\ncurl --header \"Content-Type: application/json\" \\
--header \"X-API-Key: API_KEY\" \\
--request POST \\
--data '{\"channel\": \"test\", \"data\": {\"x\": 1}}' \\
http://localhost:8000/api/publish
Note:
\n/api/publish instead of /api in pathmethod and params keys anymoreX-API-Key header for setting API key to be closer to OpenAPI specification (see more about OpenAPI below)In v5 we implemented the approach above. Many thanks to @StringNick for the help with the implementation and discussions.
\nOur HTTP and GRPC API are very similar now. We've also introduced a new batch method to send multiple commands in both HTTP and GRPC APIs, a feature that was previously unavailable in GRPC.
\nDocumentation for v5 was updated to reflect this change. But it worth noting - old API format id still supported. It will be supported for some time while we are migrating our HTTP API libraries to use modern API version. Hopefully users won't be affected by this migration a lot, just switching to a new version of library at some point.
\nOne additional benefit of moving to the new HTTP format is the possibility to define a clear OpenAPI schema for each API method Centrifugo has. It was previously quite tricky due to the fact we had one endpoint capable to work with all kinds of commands.
\nThis change paves the way for generating HTTP clients based on our OpenAPI specification.
\nWe now have Swagger UI built-in. To access it, launch Centrifugo with the \"swagger\": true option and navigate to http://localhost:8000/swagger.
The Swagger UI operates on the internal port, so if you're running Centrifugo using our Kubernetes Helm chart, it won't be exposed to the same ingress as client connection endpoints. This is similar to how our Prometheus, admin, API, and debug endpoints currently work.
\nAnother good addition is an OpenTelemetry tracing support for HTTP and GRPC server API requests. If you are using OpenTelemetry in your services you can now now enable tracing export in Centrifugo and find Centrifugo API request exported traces in your tracing collector UI.
\nDescription and simple example with Jaeger may be found in observability chapter. We only support OTLP HTTP export format and trace format defined in W3C spec: https://www.w3.org/TR/trace-context/.
\nWith the introduction of JWKS support in Centrifugo v4 (a way to validate JWT tokens using a remote endpoint which manages keys and key rotation - see JWK spec) Centrifugo users can rely on JWKS provider (like Keycloak, AWS Cognito) for making authentication.
\nBut at the same time developers may want to work with channels using subscription tokens managed in a custom way – without using the same JWKS configuration used for connection tokens.
\nCentrifugo v5 allows doing by introducing the separate_subscription_token_config option.
When separate_subscription_token_config is true Centrifugo does not look at general token options at all when verifying subscription tokens and uses config options starting from subscription_token_ prefix instead.
Here is an example how to use JWKS for connection tokens, but have HMAC-based verification for subscription tokens:
\n{
\"token_jwks_public_endpoint\": \"https://example.com/openid-connect/certs\",
\"separate_subscription_token_config\": true,
\"subscription_token_hmac_secret_key\": \"separate_secret_which_must_be_strong\"
}
With every release, Centrifugo offers more and more options. One thing we've noticed is that some options from previous Centrifugo options, which were already removed, still persist in the user's configuration file.
\nAnother issue is that a single typo in the configuration key can cost hours of debugging especially for Centrifugo new users. What is worse, the typo might result in unexpected behavior if the feature isn't properly tested before being run in production.
\nIn Centrifugo v5, we are addressing these problems. Now, Centrifugo logs on WARN level about unknown keys found in the configuration upon server start-up. Not only in the configuration file but also verifying the validity of environment variables (looking at those starting with CENTRIFUGO_ prefix). This should help clean up the configuration to align with the latest Centrifugo release and catch typos at an early stage.
It looks like this:
\n08:25:33 [WRN] unknown key found in the namespace object key=watch namespace_name=xx
08:25:33 [WRN] unknown key found in the proxy object key=type proxy_name=connect
08:25:33 [WRN] unknown key found in the configuration file key=granulr_proxy_mode
08:25:33 [WRN] unknown key found in the environment key=CENTRIFUGO_ADDRES
These warnings do not prevent server to start so you can gradually clean up the configuration.
\nCentrifugo v5 supports a special url parameter for bidirectional websocket which turns on using native WebSocket frame ping-pong mechanism instead of server-to-client application level pings Centrifugo uses by default. This simplifies debugging Centrifugo protocol with tools like Postman, wscat, websocat, etc.
\nPreviously it was inconvenient due to the fact Centrifugo sends periodic ping message to the client ({} in JSON protocol scenario) and expects pong response back within some time period. Otherwise Centrifugo closes connection. This results in problems with mentioned tools because you had to manually send {} pong message upon ping message. So typical session in wscat could look like this:
❯ wscat --connect ws://localhost:8000/connection/websocket
Connected (press CTRL+C to quit)
> {\"id\": 1, \"connect\": {}}
< {\"id\":1,\"connect\":{\"client\":\"9ac9de4e-5289-4ad6-9aa7-8447f007083e\",\"version\":\"0.0.0\",\"ping\":25,\"pong\":true}}
< {}
Disconnected (code: 3012, reason: \"no pong\")
The parameter is called cf_ws_frame_ping_pong, to use it connect to Centrifugo bidirectional WebSocket endpoint like ws://localhost:8000/connection/websocket?cf_ws_frame_ping_pong=true. Here is an example which demonstrates working with Postman WebSocket where we connect to local Centrifugo and subscribe to two channels test1 and test2:
You can then proceed to Centrifugo admin web UI, publish something to these channels and see publications in Postman.
\nNote, how we sent several JSON commands in one WebSocket frame to Centrifugo from Postman in the example above - this is possible since Centrifugo protocol supports batches of commands in line-delimited format.
\nWe consider this feature to be used only for debugging, in production prefer using our SDKs without using cf_ws_frame_ping_pong parameter – because app-level ping-pong is more efficient and our SDKs detect broken connections due to it.
As you know SockJS is deprecated in Centrifugal ecosystem since Centrifugo v4. In this release we are not removing support for it – but we may do this in the next release.
\nUnfortunately, SockJS client repo is poorly maintained these days. And some of its iframe-based transports are becoming archaic. If you depend on SockJS and you really need fallback for WebSocket – consider switching to Centrifugo own bidirectional emulation for the browser which works over HTTP-streaming (using modern fetch API with Readable streams) or SSE. It should be more performant and work without sticky sessions requirement (sticky sessions is an optimization in our implementation). More details may be found in Centrifugo v4 release post.
\nIf you think SockJS is still required for your use case - reach us out so we could think about the future steps together.
\nFinally, some important news we mentioned in the beginning!
\nCentrifugo is now backed by the company called Centrifugal Labs LTD - a Cyprus-registered technology company. This should help us to finally launch Centrifugo PRO offering – the product we have been working on for a couple of years now and which has some unique and powerful features like real-time analytics or push notification API.
\nAs a Centrifugo user you will start noticing mentions of Centrifugal Labs LTD in our licenses, Github organization, throughout this web site. And that's mostly it - no radical changes at this point. We will still be working on improving Centrifugo trying to find a balance between OSS and PRO versions. Which is difficult TBH – but we will try.
\nAn ideal plan for us – make Centrifugo development sustainable enough to have the possibility for features from the PRO version flow to the OSS version eventually. The reality may be harder than this of course.
\nThat's all about most notable things happened in Centrifugo v5. We updated documentation to reflect the changes in v5, also some documentation chapters were rewritten. For example, take a look at the refreshed Design overview. Several more changes and details outlined in the migration guide for Centifugo v5. Please feel free to contact in the community rooms if you have questions about the release. And as usual, let the Centrifugal force be with you!
", "url": "https://centrifugal.dev/blog/2023/06/29/centrifugo-v5-released", "title": "Centrifugo v5 released", "summary": "We are excited to announce a new version of Centrifugo. It's an evolutionary step which makes Centrifugo cleaner and more intuitive to use.", "date_modified": "2023-06-29T00:00:00.000Z", "author": { "name": "Centrifugal team" }, "tags": [ "centrifugo", "release" ] }, { "id": "https://centrifugal.dev/blog/2023/03/31/keycloak-sso-centrifugo", "content_html": "Securing user authentication and management can often be a challenging task when developing a modern application. As a result, many developers choose to delegate this responsibility to third-party identity providers, such as Okta, Auth0, or Keycloak.
\nIn this blog post, we'll go through the process of setting up Single Sign-On (SSO) authentication using Keycloak - popular and powerful identity provider. After setting up SSO we will create React application and connect to Centrifugo using access token generated by Keycloak for our test user:
\n\nThe integraion is possible since Centrifugo works with standard JWT for authentication and additionally supports JSON Web Key specification.
\nHere is a final source code.
\nFirst, run Keycloak using the following Docker command:
\ndocker run --rm -it -p 8080:8080 \\
-e KEYCLOAK_ADMIN=admin \\
-e KEYCLOAK_ADMIN_PASSWORD=admin \\
quay.io/keycloak/keycloak:21.0.1 start-dev
After starting Keycloak, go to http://localhost:8080/admin and login. Then perform the following tasks:
\nmyrealm.myclient. Set valid redirect URIs to http://localhost:5173/*, and web origins as http://localhost:5173.myuser and set a password for it (in Credentials tab).See this guide for additional details and illustrations of the process.
\nMake sure your created client is public (this is default) since we will request token directly from the web application.
Next, run Centrifugo using the following Docker command:
\ndocker run --rm -it -p 8000:8000 \\
-e CENTRIFUGO_ALLOWED_ORIGINS=\"http://localhost:5173\" \\
-e CENTRIFUGO_TOKEN_JWKS_PUBLIC_ENDPOINT=\"http://host.docker.internal:8080/realms/myrealm/protocol/openid-connect/certs\" \\
-e CENTRIFUGO_ALLOW_USER_LIMITED_CHANNELS=true \\
-e CENTRIFUGO_ADMIN=true \\
-e CENTRIFUGO_ADMIN_SECRET=secret \\
-e CENTRIFUGO_ADMIN_PASSWORD=admin \\
centrifugo/centrifugo:v4.1.2 centrifugo
Some comments about environment variables used here:
\nAlso note we are using host.docker.internal to access host port from inside the Docker network.
Now, let's create a new React app using very popular Vite tool:
\nnpm create vite@latest keycloak_sso_auth -- --template react
cd keycloak_sso_auth
npm install
Also, install the necessary additional packages for the React app:
\nnpm install --save @react-keycloak/web centrifuge keycloak-js
And start the development server:
\nnpm run dev
Navigate to http://localhost:5173/. You should see default Vite template working, we are going to modify it a bit.
\nUse localhost, not 127.0.0.1 - since we used localhost for Keyloak and Centrifugo configurations above.
Add the following into main.jsx:
import React from 'react'
import ReactDOM from 'react-dom/client'
import { ReactKeycloakProvider } from '@react-keycloak/web'
import App from './App'
import './index.css'
import Keycloak from \"keycloak-js\";
const keycloakClient = new Keycloak({
url: \"http://localhost:8080\",
realm: \"myrealm\",
clientId: \"myclient\"
})
ReactDOM.createRoot(document.getElementById('root')).render(
<ReactKeycloakProvider authClient={keycloakClient}>
<React.StrictMode>
<App />
</React.StrictMode>
</ReactKeycloakProvider>,
)
Note that we configured Keycloak instance pointing it to our Keycloak server. We also use @react-keycloak/web package to wrap React app into ReactKeycloakProvider component. It simplifies working with Keycloak by providing some useful hooks - we are using this hook below.
Our App component inside App.jsx may look like this:
import React, { useState, useEffect } from 'react';
import logo from './assets/centrifugo.svg'
import { Centrifuge } from \"centrifuge\";
import { useKeycloak } from '@react-keycloak/web'
import './App.css'
function App() {
const { keycloak, initialized } = useKeycloak()
if (!initialized) {
return null;
}
return (
<div>
<header>
<p>
SSO with Keycloak and Centrifugo
</p>
{keycloak.authenticated ? (
<div>
<p>Logged in as {keycloak.tokenParsed?.preferred_username}</p>
<button type=\"button\" onClick={() => keycloak.logout()}>
Logout
</button>
</div>
) : (
<button type=\"button\" onClick={() => keycloak.login()}>
Login
</button>
)}
</header>
</div >
);
}
export default App
This is actually enough for SSO flow to start working! You can click on login button and make sure that it's possible to use myuser credentials to log into the application. And log out after that.
The only missing part is Centrifugo. We can initialize connection inside useEffect hook of App component:
useEffect(() => {
if (!initialized || !keycloak.authenticated) {
return;
}
const centrifuge = new Centrifuge(\"ws://localhost:8000/connection/websocket\", {
token: keycloak.token,
getToken: function () {
return new Promise((resolve, reject) => {
keycloak.updateToken(5).then(function () {
resolve(keycloak.token);
}).catch(function (err) {
reject(err);
keycloak.logout();
});
})
}
});
centrifuge.connect();
return () => {
centrifuge.disconnect();
};
}, [keycloak, initialized]);
The important thing here is how we configure tokens: we are using Keycloak client methods to set initial token and refresh the token when required.
\nI also added some extra elements to the code to make it look a bit nicer. For example, we can listen to Centriffuge client state changes and show connection indicator on the page:
\nfunction App() {
const [connectionState, setConnectionState] = useState(\"disconnected\");
const stateToEmoji = {
\"disconnected\": \"🔴\",
\"connecting\": \"🟠\",
\"connected\": \"🟢\"
}
...
useEffect(() => {
...
centrifuge.on('state', function (ctx) {
setConnectionState(ctx.newState);
})
...
return (
...
<span className={\"connectionState \" + connectionState}>
{stateToEmoji[connectionState]}
</span>
You can find more details about Centrifugo client SDK API and states in client SDK spec.
\nIf you look at source code on Github - you will also find an example of channel subscription to a user personal channel:
\nfunction App() {
...
const [publishedData, setPublishedData] = useState(\"\");
...
useEffect(() => {
...
const userChannel = \"#\" + keycloak.tokenParsed?.sub;
const sub = centrifuge.newSubscription(userChannel);
sub.on(\"publication\", function (ctx) {
setPublishedData(JSON.stringify(ctx.data));
}).subscribe();
...
You can now:
\nThat's it! We have successfully set up Keycloak SSO authentication with Centrifugo and a React application. Again, source code is on Github.
", "url": "https://centrifugal.dev/blog/2023/03/31/keycloak-sso-centrifugo", "title": "Setting up Keycloak SSO authentication flow and connecting to Centrifugo WebSocket", "summary": "This tutorial shows how to connect to Centrifugo when using Keycloak SSO flow for user authentication. Here we build a simple demo app using React and Vite.", "date_modified": "2023-03-31T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "keycloak", "sso", "authentication" ] }, { "id": "https://centrifugal.dev/blog/2022/12/20/improving-redis-engine-performance", "content_html": "The main objective of Centrifugo is to manage persistent client connections established over various real-time transports (including WebSocket, HTTP-Streaming, SSE, WebTransport, etc – see here) and offer an API for publishing data towards established connections. Clients subscribe to channels, hence Centrifugo implements PUB/SUB mechanics to transmit published data to all online channel subscribers.
\nCentrifugo employs Redis as its primary scalability option – so that it's possible to distribute client connections amongst numerous Centrifugo nodes without worrying about channel subscribers connected to separate nodes. Redis is incredibly mature, simple, and fast in-memory storage. Due to various built-in data structures and PUB/SUB support Redis is a perfect fit to be both Centrifugo Broker and PresenceManager (we will describe what's this shortly).
In Centrifugo v4.1.0 we introduced an updated implementation of our Redis Engine (Engine in Centrifugo == Broker + PresenceManager) which provides sufficient performance improvements to our users. This post discusses the factors that prompted us to update Redis Engine implementation and provides some insight into the results we managed to achieve. We'll examine a few well-known Go libraries for Redis communication and contrast them against Centrifugo tasks.
Before we get started, let's define what Centrifugo's Broker and PresenceManager terms mean.
Broker is an interface responsible for maintaining subscriptions from different Centrifugo nodes (initiated by client connections). That helps to scale client connections over many Centrifugo instances and not worry about the same channel subscribers being connected to different nodes – since all Centrifugo nodes connected with PUB/SUB. Messages published to one node are delivered to a channel subscriber connected to another node.
\nAnother major part of Broker is keeping an expiring publication history for channels (streams). So that Centrifugo may provide a fast cache for messages missed by clients upon going offline for a short period and compensate at most once delivery of Redis PUB/SUB using Publication incremental offsets. Centrifugo uses STREAM and HASH data structures in Redis to store channel history and stream meta information.
In general Centrifugo architecture may be perfectly illustrated by this picture (Gophers are Centrifugo nodes all connected to Broker, and sockets are WebSockets):
![\"gopher-broker\"](\"https://i.imgur.com/QOJ1M9a.png\")
PresenceManager is an interface responsible for managing online presence information - list of currently active channel subscribers. While the connection is alive we periodically update presence entries for channels connection subscribed to (for channels where presence is enabled). Presence data should expire if not updated by a client connection for some time. Centrifugo uses two Redis data structures for managing presence in channels - HASH and ZSET.
\nFor a long time, the gomodule/redigo package served as the foundation for the Redis Engine implementation in Centrifugo. Huge props go to Mr Gary Burd for creating it.
\nRedigo offers a connection Pool to Redis. A simple usage of it involves getting the connection from the pool, issuing request to Redis over that connection, and then putting the connection back to the pool after receiving the result from Redis.
\nLet's write a simple benchmark which demonstrates simple usage of Redigo and measures SET operation performance:
\nfunc BenchmarkRedigo(b *testing.B) {
\tpool := redigo.Pool{
\t\tMaxIdle: 128,
\t\tMaxActive: 128,
\t\tWait: true,
\t\tDial: func() (redigo.Conn, error) {
\t\t\treturn redigo.Dial(\"tcp\", \":6379\")
\t\t},
\t}
\tdefer pool.Close()
\tb.ResetTimer()
\tb.SetParallelism(128)
\tb.ReportAllocs()
\tb.RunParallel(func(pb *testing.PB) {
\t\tfor pb.Next() {
\t\t\tc := pool.Get()
\t\t\t_, err := c.Do(\"SET\", \"redigo\", \"test\")
\t\t\tif err != nil {
\t\t\t\tb.Fatal(err)
\t\t\t}
\t\t\tc.Close()
\t\t}
\t})
}
Let's run it:
\nBenchmarkRedigo-8 228804 4648 ns/op 62 B/op 2 allocs/op
Seems pretty fast, but we can improve it further.
\nTo increase a throughput in Centrifugo, instead of using Redigo's Pool for each operation, we acquired a dedicated connection from the Pool and utilized Redis pipelining to send multiple commands where possible.
Redis pipelining improves performance by executing multiple commands using a single client-server-client round trip. Instead of executing many commands one by one, you can queue the commands in a pipeline and then execute the queued commands as if it is a single command. Redis processes commands in order and sends individual response for each command. Given a single CPU nature of Redis, reducing the number of active connections when using pipelining has a positive impact on throughput – therefore pipelining is beneficial from this angle as well.
\n![\"Redis](\"https://centrifugal.dev/assets/images/redis_pipeline-7fdadb07739a87c92e520d0fba06696d.png\")
You can quickly estimate the benefits of pipelining by running Redis locally and running redis-benchmark which comes with Redis distribution over it:
> redis-benchmark -n 100000 set key value
Summary:
throughput summary: 84674.01 requests per second
And with pipelining:
\n> redis-benchmark -n 100000 -P 64 set key value
Summary:
throughput summary: 666880.00 requests per second
In Centrifugo we are using smart batching technique for collecting pipeline (also described in one of the previous posts in this blog).
\nTo demonstrate benefits from using pipelining let's look at the following benchmark:
\nconst (
\tmaxCommandsInPipeline = 512
\tnumPipelineWorkers = 1
)
type command struct {
\terrCh chan error
}
type sender struct {
\tcmdCh chan command
\tpool redigo.Pool
}
func newSender(pool redigo.Pool) *sender {
\tp := &sender{
\t\tcmdCh: make(chan command),
\t\tpool: pool,
\t}
\tgo func() {
\t\tfor {
\t\t\tfor i := 0; i < numPipelineWorkers; i++ {
\t\t\t\tp.runPipelineRoutine()
\t\t\t}
\t\t}
\t}()
\treturn p
}
func (s *sender) send() error {
\terrCh := make(chan error, 1)
\tcmd := command{
\t\terrCh: errCh,
\t}
\t// Submit command to be executed by runPipelineRoutine.
\ts.cmdCh <- cmd
\treturn <-errCh
}
func (s *sender) runPipelineRoutine() {
\tconn := p.pool.Get()
\tdefer conn.Close()
\tfor {
\t\tselect {
\t\tcase cmd := <-s.cmdCh:
\t\t\tcommands := []command{cmd}
\t\t\tconn.Send(\"set\", \"redigo\", \"test\")
\t\tloop:
\t\t\t// Collect batch of commands to send to Redis in one RTT.
\t\t\tfor i := 0; i < maxCommandsInPipeline; i++ {
\t\t\t\tselect {
\t\t\t\tcase cmd := <-s.cmdCh:
\t\t\t\t\tcommands = append(commands, cmd)
\t\t\t\t\tconn.Send(\"set\", \"redigo\", \"test\")
\t\t\t\tdefault:
\t\t\t\t\tbreak loop
\t\t\t\t}
\t\t\t}
\t\t\t// Flush all collected commands to the network.
\t\t\terr := conn.Flush()
\t\t\tif err != nil {
\t\t\t\tfor i := 0; i < len(commands); i++ {
\t\t\t\t\tcommands[i].errCh <- err
\t\t\t\t}
\t\t\t\tcontinue
\t\t\t}
\t\t\t// Read responses to commands, they come in order.
\t\t\tfor i := 0; i < len(commands); i++ {
\t\t\t\t_, err := conn.Receive()
\t\t\t\tcommands[i].errCh <- err
\t\t\t}
\t\t}
\t}
}
func BenchmarkRedigoPipelininig(b *testing.B) {
\tpool := redigo.Pool{
\t\tWait: true,
\t\tDial: func() (redigo.Conn, error) {
\t\t\treturn redigo.Dial(\"tcp\", \":6379\")
\t\t},
\t}
\tdefer pool.Close()
\tsender := newSender(pool)
\tb.ResetTimer()
\tb.SetParallelism(128)
\tb.ReportAllocs()
\tb.RunParallel(func(pb *testing.PB) {
\t\tfor pb.Next() {
\t\t\terr := sender.send()
\t\t\tif err != nil {
\t\t\t\tb.Fatal(err)
\t\t\t}
\t\t}
\t})
}
This is a strategy that we employed in Centrifugo for a long time. As you can see code with automatic pipelining gets more complex, and in real life it's even more complicated to support different types of commands, channel send timeouts, and server shutdowns.
\nWhat about the performance of this approach?
\nBenchmarkRedigo-8 228804 4648 ns/op 62 B/op 2 allocs/op
BenchmarkRedigoPipelininig-8 1840758 604.7 ns/op 176 B/op 4 allocs/op
Operation latency reduced from 4648 ns/op to 604.7 ns/op – not bad right?
\nIt's worth mentioning that upon increased RTT between application and Redis the approach with pipelining will provide worse throughput. But it still can be better than in pool-based approach. Let's say we have latency 5ms between app and Redis. This means that with pool size of 128 you will be able to issue up to 128 * (1000 / 5) = 25600 requests per second over 128 connections. With the pipelining approach above the theoretical limit is 512 * (1000 / 5) = 102400 requests per second over a single connection (though in case of using code for pipelining shown above we need to have larger parallelism, say 512 instead of 128). And it can scale further if you increase numPipelineWorkers to work over several connections in paralell. Though increasing numPipelineWorkers has negative effect on CPU – we will discuss this later in this post.
Redigo is an awesome battle-tested library that served us great for a long time.
\nThere are three modes in which Centrifugo can work with Redis these days:
\nAll modes additionally can be used with client-side consistent sharding. So it's possible to scale Redis even without a Redis Cluster setup.
\nUnfortunately, with pure Redigo library, it's only possible to implement [ 1 ] – i.e. connecting to a single standalone Redis instance.
\nTo support the scheme with Sentinel you whether need to have a proxy between the application and Redis which proxies the connection to Redis master. For example, with Haproxy it's possible in this way:
\nlisten redis
server redis-01 127.0.0.1:6380 check port 6380 check inter 2s weight 1 inter 2s downinter 5s rise 10 fall 2 on-marked-down shutdown-sessions on-marked-up shutdown-backup-sessions
server redis-02 127.0.0.1:6381 check port 6381 check inter 2s weight 1 inter 2s downinter 5s rise 10 fall 2 backup
bind *:6379
mode tcp
option tcpka
option tcplog
option tcp-check
tcp-check send PING\\r\\n
tcp-check expect string +PONG
tcp-check send info\\ replication\\r\\n
tcp-check expect string role:master
tcp-check send QUIT\\r\\n
tcp-check expect string +OK
balance roundrobin
Or, you need to additionally import FZambia/sentinel library - which provides a communication layer with Redis Sentinel on top of Redigo's connection Pool.
\nFor communicating with Redis Cluster one more library may be used – mna/redisc which is also a layer on top of redigo basic functionality.
Combining redigo + FZambia/sentinel + mna/redisc we managed to implement all three connection modes. This worked, though resulted in rather tricky Redis setup. Also, it was difficult to re-use existing pipelining code we had for a standalone Redis with Redis Cluster. As a result, Centrifugo only used pipelining in a standalone or Sentinel Redis cases. When using Redis Cluster, however, Centrifugo merely used the connection pool to issue requests thus not benefiting from request pipelining. Due to this we had some code duplication to send the same requests in various Redis configurations.
Another thing is that Redigo uses interface{} for command construction. To send command to Redis Redigo has Do method which accepts name of the command and variadic interface{} arguments to construct command arguments:
Do(commandName string, args ...interface{}) (reply interface{}, err error)
While this works well and you can issue any command to Redis, you need to be very accurate when constructing a command. This also adds some allocation overhead. As we know more memory allocations lead to the increased CPU utilization because the allocation process itself requires more processing power and the GC is under more strain.
\nAt some point we felt that eliminating additional dependencies (even though I am the author of one of them) and reducing allocations in Redis communication layer is a nice step forward for Centrifugo. So we started looking around for redigo alternatives.
To summarize, here is what we wanted from Redis library:
\nThe most obvious alternative to Redigo is go-redis/redis package. It's popular, regularly gets updates, used by a huge amount of Go projects (Grafana, Thanos, etc.). And maintained by\nVladimir Mihailenco who created several more awesome Go libraries, like msgpack for example. I personally successfully used go-redis/redis in several other projects I worked on.
To avoid setup boilerplate for various Redis installation variations go-redis/redis has UniversalClient. From docs:
\n\nUniversalClient is a wrapper client which, based on the provided options, represents either a ClusterClient, a FailoverClient, or a single-node Client. This can be useful for testing cluster-specific applications locally or having different clients in different environments.
\n
In terms of implementation go-redis/redis also has internal pool of connections to Redis, similar to redigo. It's also possible to use Client.Pipeline method to allocate a Pipeliner interface and use it for pipelining. So UniversalClient reduces setup boilerplate for different Redis installation types and number of dependencies we had, and it provide very similar way to pipeline requests so we could easily re-implement things we had with Redigo.
Go-redis also provides more type-safety when constructing commands compared to Redigo, almost every command in Redis is implemented as a separate method of Client, for example Publish defined as:
func (c Client) Publish(ctx context.Context, channel string, message interface{}) *IntCmd
You can see though that we still have interface{} here for message argument type. I suppose this was implemented in such way for convenience – to pass both string or []byte. But it still produces some extra allocations.
Without pipelining the simplest program with go-redis/redis may look like this:
func BenchmarkGoredis(b *testing.B) {
\tclient := redis.NewUniversalClient(&redis.UniversalOptions{
\t\tAddrs: []string{\":6379\"},
\t\tPoolSize: 128,
\t})
\tdefer client.Close()
\tb.ResetTimer()
\tb.SetParallelism(128)
\tb.ReportAllocs()
\tb.RunParallel(func(pb *testing.PB) {
\t\tfor pb.Next() {
\t\t\tresp := client.Set(context.Background(), \"goredis\", \"test\", 0)
\t\t\tif resp.Err() != nil {
\t\t\t\tb.Fatal(resp.Err())
\t\t\t}
\t\t}
\t})
}
Let's run it:
\nBenchmarkRedigo-8 228804 4648 ns/op 62 B/op 2 allocs/op
BenchmarkGoredis-8 268444 4561 ns/op 244 B/op 8 allocs/op
Result is pretty comparable to Redigo, though Go-redis allocates more (btw most of allocations come from the connection liveness check upon getting from the pool which can not be turned off).
\n![](\"https://centrifugal.dev/assets/images/goredis_allocs-3b6ad3ecb2543d679e812619438962c3.png\")
It's interesting – if we dive deeper into what is it we can discover that this is the only way in Go to check connection was closed without reading data from it. The approach was originally introduced by go-sql-driver/mysql, it's not cross-platform, and related issue may be found in Go issue tracker.
\nBut as I said in Centrifugo we already used pipelining over the dedicated connection for all operations so we avoid frequently getting connections from the pool. And early experiments proved that go-redis may provide some performance benefits for our use case.
At some point @j178 sent a pull request to Centrifuge library with Broker and PresenceManager implementations based on go-redis/redis. The amount of code to cover all the various Redis setups was reduced, we got only one dependency instead of three 🔥
But what about performance? Here we will show results for several operations which are typical for Centrifugo:
\nRedisPublish)RedisPublish_History)RedisSubscribe)RedisRecover).RedisAddPresence)Here are the benchmark results we got when comparing redigo (v1.8.9) implementation (old) and go-redis/redis (v9.0.0-rc.2) implementation (new) with Redis v6.2.7 on Mac with M1 processor and benchmark paralellism 128:
❯ benchstat redigo_p128.txt goredis_p128.txt
name old time/op new time/op delta
RedisPublish-8 1.45µs ±10% 1.88µs ± 4% +29.32% (p=0.000 n=10+10)
RedisPublish_History-8 12.5µs ± 6% 9.7µs ± 3% -22.77% (p=0.000 n=10+10)
RedisSubscribe-8 1.47µs ±24% 1.47µs ±10% ~ (p=0.469 n=10+10)
RedisRecover-8 18.4µs ± 2% 6.3µs ± 0% -65.78% (p=0.000 n=10+8)
RedisAddPresence-8 3.72µs ± 1% 3.40µs ± 1% -8.74% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
RedisPublish-8 483B ± 0% 499B ± 0% +3.37% (p=0.000 n=9+10)
RedisPublish_History-8 1.30kB ± 0% 1.08kB ± 0% -16.67% (p=0.000 n=10+10)
RedisSubscribe-8 892B ± 2% 662B ± 6% -25.83% (p=0.000 n=10+10)
RedisRecover-8 1.25kB ± 1% 1.00kB ± 0% -19.91% (p=0.000 n=10+10)
RedisAddPresence-8 907B ± 0% 827B ± 0% -8.82% (p=0.002 n=7+8)
name old allocs/op new allocs/op delta
RedisPublish-8 10.0 ± 0% 9.0 ± 0% -10.00% (p=0.000 n=10+10)
RedisPublish_History-8 29.0 ± 0% 25.0 ± 0% -13.79% (p=0.000 n=10+10)
RedisSubscribe-8 22.0 ± 0% 14.0 ± 0% -36.36% (p=0.000 n=8+7)
RedisRecover-8 29.0 ± 0% 23.0 ± 0% -20.69% (p=0.000 n=10+10)
RedisAddPresence-8 18.0 ± 0% 17.0 ± 0% -5.56% (p=0.000 n=10+10)
Please note that this benchmark is not a pure performance comparison of two Go libraries for Redis – it's a performance comparison of Centrifugo Engine methods upon switching to a new library.
Or visualized in Grafana:
\n![](\"https://centrifugal.dev/assets/images/redis_vis01-8364cd7474502a278b9362d5f5e71245.png\")
Centrifugo benchmarks results shown in the post use parallelism 128. If someone interested to check numbers for paralellism 1 or 16 – check out this comment on Github.
We observe a noticeable reduction in allocations in these benchmarks and in most benchmarks (presented here and other not listed in this post) we observed a reduced latency.
\nOverall, results convinced us that the migration from redigo to go-redis/redis may provide Centrifugo with everything we aimed for – all the goals for a redigo alternative outlined above were successfully fullfilled.
One good thing go-redis/redis allowed us to do is to use Redis pipelining also in a Redis Cluster case. It's possible due to the fact that go-redis/redis re-maps pipeline objects internally based on keys to execute pipeline on the correct node of Redis Cluster. Actually, we could do the same based on redigo + mna/redisc, but here we got it for free.
BTW, there is a page with comparison between redigo and go-redis/redis in go-redis/redis docs which outlines some things I mentioned here and some others.
But we have not migrated to go-redis/redis in the end. And the reason is another library – rueidis.
While results were good with go-redis/redis we also made an attempt to implement Redis Engine on top of rueian/rueidis library written by @rueian. According to docs, rueidis is:
\n\nA fast Golang Redis client that supports Client Side Caching, Auto Pipelining, Generics OM, RedisJSON, RedisBloom, RediSearch, RedisAI, RedisGears, etc.
\n
The readme of rueidis contains benchmark results where it hugely outperforms go-redis/redis in terms of operation latency/throughput in both single Redis and Redis Custer setups:
![](\"https://centrifugal.dev/assets/images/rueidis_1-05b9a03175ec64b4ec70679313bd4a38.png\")
![](\"https://centrifugal.dev/assets/images/rueidis_2-55b7b75cf531852fed8f73ceb7883b4d.png\")
rueidis works with standalone Redis, Sentinel Redis and Redis Cluster out of the box. Just like UniversalClient of go-redis/redis. So it also allowed us to reduce code boilerplate to work with all these setups.
Again, let's try to write a simple program like we had for Redigo and Go-redis above:
\nfunc BenchmarkRueidis(b *testing.B) {
\tclient, err := rueidis.NewClient(rueidis.ClientOption{
\t\tInitAddress: []string{\":6379\"},
\t})
\tif err != nil {
\t\tb.Fatal(err)
\t}
\tb.ResetTimer()
\tb.SetParallelism(128)
\tb.ReportAllocs()
\tb.RunParallel(func(pb *testing.PB) {
\t\tfor pb.Next() {
\t\t\tcmd := client.B().Set().Key(\"rueidis\").Value(\"test\").Build()
\t\t\tres := client.Do(context.Background(), cmd)
\t\t\tif res.Error() != nil {
\t\t\t\tb.Fatal(res.Error())
\t\t\t}
\t\t}
\t})
}
And run it:
\nBenchmarkRedigo-8 228804 4648 ns/op 62 B/op 2 allocs/op
BenchmarkGoredis-8 268444 4561 ns/op 244 B/op 8 allocs/op
BenchmarkRueidis-8 2908591 418.5 ns/op 4 B/op 1 allocs/op
rueidis library comes with automatic implicit pipelining, so you can send each request in isolated way while rueidis makes sure the request becomes part of the pipeline sent to Redis – thus utilizing the connection between an application and Redis most efficiently with maximized throughput. The idea of implicit pipelining with Redis is not new and Go ecosystem already had joomcode/redispipe library which implemented it (though it comes with some limitations which made it unsuitable for Centrifugo use case).
So applications that use a pool-based approach for communication with Redis may observe dramatic improvements in latency and throughput when switching to the Rueidis library.
\nFor Centrifugo we didn't expect such a huge speed-up as shown in the above graphs since we already used pipelining in Redis Engine. But rueidis implements some ideas which allow it to be efficient. Insights about these ideas are provided by Rueidis author in a \"Writing a High-Performance Golang Client Library\" series of posts on Medium:
I did some prototypes with rueidis which were super-promising in terms of performance. There were some issues found during that early prototyping (mostly with PUB/SUB) – but all of them were quickly resolved by Rueian.
Until v0.0.80 release rueidis did not support RESP2 though, so we could not replace our Redis Engine implementation with it. But as soon as it got RESP2 support we opened a pull request with alternative implementation.
Since auto-pipelining is used in rueidis by default we were able to remove some of our own pipelining management code – so the Engine implementation is more concise now. One more thing to mention is a simpler PUB/SUB code we were able to write with rueidis. One example is that in redigo case we had to periodically PING PUB/SUB connection to maintain it alive, rueidis does this automatically.
Regarding performance, here are the benchmark results we got when comparing redigo (v1.8.9) implementation (old) and rueidis (v0.0.90) implementation (new):
❯ benchstat redigo_p128.txt rueidis_p128.txt
name old time/op new time/op delta
RedisPublish-8 1.45µs ±10% 0.56µs ± 1% -61.53% (p=0.000 n=10+9)
RedisPublish_History-8 12.5µs ± 6% 9.7µs ± 1% -22.43% (p=0.000 n=10+9)
RedisSubscribe-8 1.47µs ±24% 1.45µs ± 1% ~ (p=0.484 n=10+9)
RedisRecover-8 18.4µs ± 2% 6.2µs ± 1% -66.08% (p=0.000 n=10+10)
RedisAddPresence-8 3.72µs ± 1% 3.60µs ± 1% -3.34% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
RedisPublish-8 483B ± 0% 91B ± 0% -81.16% (p=0.000 n=9+10)
RedisPublish_History-8 1.30kB ± 0% 0.39kB ± 0% -70.08% (p=0.000 n=10+8)
RedisSubscribe-8 892B ± 2% 360B ± 0% -59.66% (p=0.000 n=10+10)
RedisRecover-8 1.25kB ± 1% 0.36kB ± 1% -71.52% (p=0.000 n=10+10)
RedisAddPresence-8 907B ± 0% 151B ± 1% -83.34% (p=0.000 n=7+9)
name old allocs/op new allocs/op delta
RedisPublish-8 10.0 ± 0% 2.0 ± 0% -80.00% (p=0.000 n=10+10)
RedisPublish_History-8 29.0 ± 0% 10.0 ± 0% -65.52% (p=0.000 n=10+10)
RedisSubscribe-8 22.0 ± 0% 6.0 ± 0% -72.73% (p=0.002 n=8+10)
RedisRecover-8 29.0 ± 0% 7.0 ± 0% -75.86% (p=0.000 n=10+10)
RedisAddPresence-8 18.0 ± 0% 3.0 ± 0% -83.33% (p=0.000 n=10+10)
Or visualized in Grafana:
\n![](\"https://centrifugal.dev/assets/images/redis_vis02-a8e078e75bac446b0c907beeeaf368ae.png\")
2.5x times more publication throughput than we had before! Instead of 700k publications/sec, we went towards 1.7 million publications/sec due to drastically decreased publish operation latency (1.45µs -> 0.59µs). This means that our previous Engine implementation under-utilized Redis, and Rueidis just pushes us towards Redis limits. The latency of most other operations is also reduced.
\nThe allocation effectiveness of the implementation based on \"rueidis\" is best. As you can see rueidis helped us to generate sufficiently fewer memory allocations for all our Redis operations. Allocation improvements directly affect Centrifugo node CPU usage. Though we will talk about CPU more later below.
For Redis Cluster case we also got benchmark results similar to the standalone Redis results above.
\nI might add that I enjoyed building commands with rueidis. All Redis commands may be constructed using a builder approach. Rueidis comes with builders generated for all Redis commands. As an illustration, this is a process of building a PUBLISH Redis command:
This drastically reduces a chance to make a stupid mistake while constructing a command. Instead of always opening Redis docs to see a command syntax it's now possible to just start typing - and quickly come to the complete command to send.
\nAfter making all these benchmarks and implementing Engine in Rueidis I decided to check whether Centrifugo consumes less CPU with it. I expected a notable CPU reduction as Rueidis Engine implementation allocates much less than Redigo-based. Turned out it's not that simple.
\nI ran Centrifugo with some artificial load and noticed that CPU consumption of the new implementation is actually... worse than we had with Redigo-based engine under equal conditions!😩 But why?
\nAs I mentioned above Redis pipelining is a technique when several commands may be combined into one batch to send over the network. In case of automatic pipelining the size of generated batches start playing a crucial role in application and Redis CPU usage – since smaller command batches result into more read/write system calls to the kernel on both application and Redis server sides. That's why projects like Twemproxy which sit between app and Redis have sich a good effect on Redis CPU usage among other things.
\nAs we have seen above, Rueidis provides a better throughput and latency, but it's more agressive in terms of flushing data to the network. So in its default configuration we get smaller batches under th equal conditions than we had before in our own pipelining implementation based on Redigo (shown in the beginning of this post).
\nLuckily, there is an option in Rueidis called MaxFlushDelay which allows to slow down write loop a bit to give Rueidis a chance to collect more commands to send in one batch. When this option is used Rueidis will make a pause after each network flush not bigger than selected value of MaxFlushDelay (please note, that this is a delay after flushing collected pipeline commands, not an additional delay for each request). Using some reasonable value it's possible to drastically reduce both application and Redis CPU utilization.
To demonstrate this I created a repo: https://github.com/FZambia/pipelines.
\nThis repo contains three benchmarks where we use automatic pipelining: based on redigo, based on go-redis/redis and rueidis. In these benchmarks we produce concurrent requests, but instead of pushing the system towards the limits we are limiting number of requests sent to Redis, so we put all libraries in equal conditions.
To rate limit requests we are using uber-go/ratelimit library. For example, to allow rate no more than 100k commands per second we can do sth like this:
\nrl := ratelimit.New(100, ratelimit.Per(time.Millisecond))
for {
\trl.Take()
\t...
}
We limit requests per second we could actually just write ratelimit.New(100000) – but we aim to get a more smooth distribution of requests over time - so using millisecond resolution.
Let's run all the benchmarks in the default configuration:
\n\nAverage CPU usage during the test (a bit rough but enough for demonstration):
\n| Redigo | Go-redis/redis | Rueidis | |
|---|---|---|---|
| Application CPU, % | 95 | 99 | 116 |
| Redis CPU, % | 36 | 35 | 42 |
OK, Rueidis-based implementation is the worst here despite of allocating less than others. So let's try to change this by setting MaxFlushDelay to sth like 100 microseconds:
Now CPU usage is:
\n| Redigo | Go-redis/redis | Rueidis | |
|---|---|---|---|
| Application CPU, % | 95 | 99 | 59 |
| Redis CPU, % | 36 | 35 | 12 |
So we can achieve great CPU usage reduction. CPU went from 116% to 59% for the application side, and from 42% to only 12% for Redis! We are sacrificing latency though. Given the fact the CPU utilization reduction is very notable the trade-off is pretty fair.
\nIt's definitely possible to improve CPU usage in Redigo and Go-redis/redis cases too – using similar technique. But the goal here was to improve Rueidis-based engine implementation to make it comparable or better than our Redigo-based implementation in terms of CPU utilization.
As you can see we were able to achieve better CPU results just by using 100 microseconds delay after each network flush. In real life, where we are not running Redis on localhost and have some network latency in between application and Redis, this delay should be insignificant at all. Indeed, adding MaxFlushDelay can even improve (!) the latency you have. You may wonder what happened with benchmarks we showed above after we added MaxFlushDelay option. In Centrifugo we chose default value 100 microseconds, and here are results on localhost (old without delay, new with delay):
> benchstat rueidis_p128.txt rueidis_delay_p128.txt
name old time/op new time/op delta
RedisPublish-8 559ns ± 1% 468ns ± 0% -16.35% (p=0.000 n=9+8)
RedisPublish_History-8 9.72µs ± 1% 9.67µs ± 1% -0.52% (p=0.007 n=9+8)
RedisSubscribe-8 1.45µs ± 1% 1.27µs ± 1% -12.49% (p=0.000 n=9+10)
RedisRecover-8 6.25µs ± 1% 5.85µs ± 0% -6.32% (p=0.000 n=10+10)
RedisAddPresence-8 3.60µs ± 1% 3.33µs ± 1% -7.52% (p=0.000 n=10+10)
(rest is not important here...)
It's even better for this set of benchmarks. Though while it's better for these benchmarks the numbers may differ for other under different conditions. For example, in the benchmarks we run we use concurrency 128, if we reduce concurrency we will notice reduced throughput – as batches Rueidis collects become smaller. Smaller batches + some delay to collect = less requests per second to send.
\nThe problem is that the value to pause Rueidis write loop is a very use case specific, it's pretty hard to provide a reasonable default for it. Depending on request rate/size, network latency etc. you may choose a larger or smaller delay. In v4.1.0 we start with hardcoded 100 microsecond MaxFlushDelay which seems sufficient for most use cases and showed good results in the benchmarks - though possibly we will have to make it tunable later.
To check that Centrifugo benchmarks also utilize less CPU I added rate limiter (50k rps per second) to benchmarks and compared version without MaxFlushDelay and with 100 microsecond MaxFlushDelay:
| 50k req per second | Without delay | With 100mks delay |
|---|---|---|
| BenchmarkPublish | Centrifugo - 75%, Redis - 24% | Centrifugo - 44%, Redis - 9% |
| BenchmarkPublish_History | Centrifugo - 80% , Redis - 67% | Centrifugo - 55%, Redis - 50% |
| BenchmarkSubscribe | Centrifugo - 80%, Redis - 30% | Centrifugo - 45% , Redis - 14% |
| BenchmarkRecover | Centrifugo - 84%, Redis - 51% | Centrifugo - 51%, Redis - 36% |
| BenchmarkPresence | Centrifugo - 114%, Redis - 69% | Centrifugo - 90%, Redis - 60% |
In this test I replaced BenchmarkAddPresence with BenchmarkPresence (get information about all online subscribers in channel) to also make sure we have CPU reduction when using read-intensive method, i.e. when Redis response is reasonably large.
We observe a notable CPU usage improvement here.
\nHope you understand now why increasing numPipelineWorkers value in the pipelining code showed before results into increased CPU usage on app and Redis sides – due to smaller batch sizes and more read/write system calls as the consequence.
BTW, would it be a nice thing if Go benchmarking suite could show a CPU usage of the process in addition to time and alloc stats? 🤔
The last thing to check is how new implementation works upon increased RTT between application and Redis. To add artificial latency on localhost on Linux one can use tc tool as shown here by Daniel Stenberg. But I am on MacOS so the simplest way I found was using Shopify/toxiproxy. Sth like running a server:
toxyproxy-server
And then in another terminal I used toxiproxy-cli to create toxic Redis proxy with additional latency on port 26379:
toxiproxy-cli create -l localhost:26379 -u localhost:6379 toxic_redis
toxiproxy-cli toxic add -t latency -a latency=5 toxic_redis
The benchmark results are (old is Redigo-based, new is Rueidis-based):
> benchstat redigo_latency_p128.txt rueidis_delay_latency_p128.txt
name old time/op new time/op delta
RedisPublish-8 31.5µs ± 1% 5.6µs ± 3% -82.26% (p=0.000 n=9+10)
RedisPublish_History-8 62.8µs ± 3% 10.6µs ± 4% -83.05% (p=0.000 n=10+10)
RedisSubscribe-8 1.52µs ± 5% 6.05µs ± 8% +298.70% (p=0.000 n=8+10)
RedisRecover-8 48.3µs ± 3% 7.3µs ± 4% -84.80% (p=0.000 n=10+10)
RedisAddPresence-8 52.3µs ± 4% 5.8µs ± 2% -88.94% (p=0.000 n=10+10)
(rest is not important here...)
We see that new Engine implementation behaves much better for most cases. But what happened to Subscribe operation? It did not change at all in Redigo case – the same performance as if there is no additional latency involved!
Turned out that when we call Subscribe in Redigo case, Redigo only flushes data to the network without waiting synchronously for subscribe result.
It makes sense in general and we can listen to subscribe notifications asynchronously, but in Centrifugo we relied on the returned error thinking that it includes succesful subscription result from Redis - meaning that we already subscribed to a channel at that point. And this could theoretically lead to some rare bugs in Centrifugo.
\nRueidis library waits for subscribe response. So here the behavior of rueidis while differs from redigo in terms of throughput under increased latency just fits Centrifugo better in terms of behavior. So we go with it.
Migrating from Redigo to Rueidis library was not just a task of rewriting code, we had to carefully test various aspects of Redis Engine behaviour – latency, throughput, CPU utilization of application, and even CPU utilization of Redis itself under the equal application load conditions.
\nI think that we will find more projects in Go ecosystem using rueidis library shortly. Not just because of its allocation efficiency and out-of-the-box throughput, but also due to a convenient type-safe command API.
For most Centrifugo users this migration means more efficient CPU usage as new implementation allocates less memory (less work to allocate and less strain on GC) and we tried to find a reasonable batch size to reduce the number of system calls for common operations. While latency and throughput of single Centrifugo node should be better as we make concurrent Redis calls from many goroutines.
\nHopefully readers will learn some tips from this post which can help to achieve effective communication with Redis from Go or another programming language.
\nA few key takeaways:
\nP.S. One thing worth mentioning and which may be helpful for someone is that during our comparison experiments we discovered that Redis 7 has a major latency increase compared to Redis 6 when executing Lua scripts. So if you have performance sensitive code with Lua scripts take a look at this Redis issue. With the help of Redis developers some things already improved in unstable Redis branch, hopefully that issue will be closed at the time you read this post.
Let's say you develop an application and want a real-time connection which is subscribed to one channel. Let's also assume that this channel is used for user personal notifications. So only one user in the application can subcribe to that channel to receive its notifications in real-time.
\nIn this post we will look at various ways to achieve this with Centrifugo, and consider trade-offs of the available approaches. The main goal of this tutorial is to help Centrifugo newcomers be aware of all the ways to control channel permissions by reading just one document.
\nAnd... well, there are actually 8 ways I found, not 101 😇
\nTo make the post a bit easier to consume let's setup some things. Let's assume that the user for which we provide all the examples in this post has ID \"17\". Of course in real-life the examples given here can be extrapolated to any user ID.
When you create a real-time connection to Centrifugo the connection is authenticated using the one of the following ways:
\nAs soon as the connection is successfully established and authenticated Centrifugo knows the ID of connected user. This is important to understand.
\nAnd let's define a namespace in Centrifugo configuration which will be used for personal user channels:
\n{
...
\"namespaces\": [
{
\"name\": \"personal\",
\"presence\": true
}
]
}
Defining namespaces for each new real-time feature is a good practice in Centrifugo. As an awesome improvement we also enabled presence in the personal namespace, so whenever users subscribe to a channel in this namespace Centrifugo will maintain online presence information for each channel. So you can find out all connections of the specific user existing at any moment. Defining presence is fully optional though - turn it of if you don't need presence information and don't want to spend additional server resources on maintaining presence.
Probably the most performant approach.
All you need to do is to extend namespace configuration with allow_user_limited_channels option:
{
\"namespaces\": [
{
\"name\": \"personal\",
\"presence\": true,
\"allow_user_limited_channels\": true
}
]
}
On the client side you need to have sth like this (of course the ID of current user will be dynamic in real-life):
\nconst sub = centrifuge.newSubscription('personal:#17');
sub.on('publication', function(ctx) {
console.log(ctx.data);
})
sub.subscribe();
Here you are subscribing to a channel in personal namespace and listening to publications coming from a channel. Having # in channel name tells Centrifugo that this is a user-limited channel (because # is a special symbol that is treated in a special way by Centrifugo as soon as allow_user_limited_channels enabled).
In this case the user ID part of user-limited channel is \"17\". So Centrifugo allows user with ID \"17\" to subscribe on personal:#17 channel. Other users won't be able to subscribe on it.
To publish updates to subscription all you need to do is to publish to personal:#17 using server publish API (HTTP or GRPC).
Probably the most flexible approach, with reasonably good performance characteristics.
Another way we will look at is using subscription JWT for subscribing. When you create Subscription object on the client side you can pass it a subscription token, and also provide a function to retrieve subscription token (useful to automatically handle token refresh, it also handles initial token loading).
\nconst token = await getSubscriptionToken('personal:17');
const sub = centrifuge.newSubscription('personal:17', {
token: token
});
sub.on('publication', function(ctx) {
console.log(ctx.data);
})
sub.subscribe();
Inside getSubscriptionToken you can issue a request to the backend, for example in browser it's possible to do with fetch API.
On the backend side you know the ID of current user due to the native session mechanism of your app, so you can decide whether current user has permission to subsribe on personal:17 or not. If yes – return subscription JWT according to our rules. If not - return empty string so subscription will go to unsubscribed state with unauthorized reason.
Here are examples for generating subscription HMAC SHA-256 JWTs for channel personal:17 and HMAC secret key secret:
import jwt
import time
claims = {
\"sub\": \"17\",
\"channel\": \"personal:17\"
\"exp\": int(time.time()) + 30*60
}
token = jwt.encode(claims, \"secret\", algorithm=\"HS256\").decode()
print(token)
const jose = require('jose')
(async function main() {
const secret = new TextEncoder().encode('secret')
const alg = 'HS256'
const token = await new jose.SignJWT({ 'sub': '17', 'channel': 'personal:17' })
.setProtectedHeader({ alg })
.setExpirationTime('30m')
.sign(secret)
console.log(token);
})();
Since we set expiration time for subscription JWT tokens we also need to provide a getToken function to a client on the frontend side:
const sub = centrifuge.newSubscription('personal:17', {
getToken: async function (ctx) {
const token = await getSubscriptionToken('personal:17');
return token;
}
});
sub.on('publication', function(ctx) {
console.log(ctx.data);
})
sub.subscribe();
This function will be called by SDK automatically to refresh subscription token when it's going to expire. And note that we omitted setting token option here – since SDK is smart enough to call provided getToken function to extract initial subscription token from the backend.
The good thing in using subscription JWT approach is that you can provide token expiration time, so permissions to subscribe on a channel will be validated from time to time while connection is active. You can also provide additional channel context info which will be attached to presence information (using info claim of subscription JWT). And you can granularly control channel permissions using allow claim of token – and give client capabilities to publish, call history or presence information (this is Centrifugo PRO feature at this point). Token also allows to override some namespace options on per-subscription basis (with override claim).
Using subscription tokens is a general approach for any channels where you need to check access first, not only for personal user channels.
\nProbably the most secure approach.
Subscription JWT gives client a way to subscribe on a channel, and avoid requesting your backend for permission on every resubscribe. Token approach is very good in massive reconnect scenario, when you have many connections and they all resubscribe at once (due to your load balancer reload, for example). But this means that if you unsubscribed client from a channel using server API, client can still resubscribe with token again - until token will expire. In some cases you may want to avoid this.
\nAlso, in some cases you want to be notified when someone subscribes to a channel.
\nIn this case you may use subscribe proxy feature. When using subscribe proxy every attempt of a client to subscribe on a channel will be translated to request (HTTP or GRPC) from Centrifugo to the application backend. Application backend can decide whether client is allowed to subscribe or not.
\nOne advantage of using subscribe proxy is that backend can additionally provide initial channel data for the subscribing client. This is possible using data field of subscribe result generated by backend subscribe handler.
{
\"proxy_subscribe_endpoint\": \"http://localhost:9000/centrifugo/subscribe\",
\"namespaces\": [
{
\"name\": \"personal\",
\"presence\": true,
\"proxy_subscribe\": true
}
]
}
And on the backend side define a route /centrifugo/subscribe, check permissions of user upon subscription and return result to Centrifugo according to our subscribe proxy docs. Or simply run GRPC server using our proxy definitions and react on subscription attempt sent from Centrifugo to backend over GRPC.
On the client-side code is as simple as:
\nconst sub = centrifuge.newSubscription('personal:17');
sub.on('publication', function(ctx) {
console.log(ctx.data);
})
sub.subscribe();
The approach where you don't need to manage client-side subscriptions.
Server-side subscriptions is a way to consume publications from channels without even create Subscription objects on the client side. In general, client side Subscription objects provide a more flexible and controllable way to work with subscriptions. Clients can subscribe/unsubscribe on channels at any point. Client-side subscriptions provide more details about state transitions.
\nWith server-side subscriptions though you are consuming publications directly from Client instance:
\nconst client = new Centrifuge('ws://localhost:8000/connection/websocket', {
token: 'CONNECTION-JWT'
});
client.on('publication', function(ctx) {
console.log('publication received from server-side channel', ctx.channel, ctx.data);
});
client.connect();
In this case you don't have separate Subscription objects and need to look at ctx.channel upon receiving publication or to publication content to decide how to handle it. Server-side subscriptions could be a good choice if you are using Centrifugo unidirectional transports and don't need dynamic subscribe/unsubscribe behavior.
The first way to subscribe client on a server-side channel is to include channels claim into connection JWT:
{
\"sub\": \"17\",
\"channels\": [\"personal:17\"]
}
Upon successful connection user will be subscribed to a server-side channel by Centrifugo. One downside of using server-side channels is that errors in one server-side channel (like impossible to recover missed messages) may affect the entire connection and result into reconnects, while with client-side subscriptions individual subsription failures do not affect the entire connection.
\nBut having one server-side channel per-connection seems a very reasonable idea to me in many cases. And if you have stable set of subscriptions which do not require lifetime state management – this can be a nice approach without additional protocol/network overhead involved.
\nSimilar to the previous one for cases when you are authenticating connections over connect proxy instead of using JWT.
\nThis is possible using channels field of connect proxy handler result. The code on the client-side is the same as in Option #4 – since we only change the way how list of server-side channels is provided.
Almost no code approach.
As we pointed above Centrifugo knows an ID of the user due to authentication process. So why not combining this knowledge with automatic server-side personal channel subscription? Centrifugo provides exactly this with user personal channel feature.
\n{
\"user_subscribe_to_personal\": true,
\"user_personal_channel_namespace\": \"personal\",
\"namespaces\": [
{
\"name\": \"personal\",
\"presence\": true
}
]
}
This feature only subscribes non-anonymous users to personal channels (those with non-empty user ID). The configuration above will subscribe our user \"17\" to channel personal:#17 automatically after successful authentication.
Allows using client-side subscriptions, but skip receiving subscription token. This is only available in Centrifugo PRO at this point.
\nSo when generating JWT you can provide additional caps claim which contains channel resource capabilities:
import jwt
import time
claims = {
\"sub\": \"17\",
\"exp\": int(time.time()) + 30*60,
\"caps\": [
{
\"channels\": [\"personal:17\"],
\"allow\": [\"sub\"]
}
]
}
token = jwt.encode(claims, \"secret\", algorithm=\"HS256\").decode()
print(token)
const jose = require('jose');
(async function main() {
const secret = new TextEncoder().encode('secret')
const alg = 'HS256'
const token = await new jose.SignJWT({
sub: '17',
caps: [
{
\"channels\": [\"personal:17\"],
\"allow\": [\"sub\"]
}
]
})
.setProtectedHeader({ alg })
.setExpirationTime('30m')
.sign(secret)
console.log(token);
})();
While in case of single channel the benefit of using this approach is not really obvious, it can help when you are using several channels with stric access permissions per connection, where providing capabilities can help to save some traffic and CPU resources since we avoid generating subscription token for each individual channel.
\nThis is very similar to the previous approach, but capabilities are passed to Centrifugo in connect proxy result. So if you are using connect proxy for auth then you can still provide capabilities in the same form as in JWT. This is also a Centrifugo PRO feature.
\nWhich way to choose? Well, it depends. Since your application will have more than only a personal user channel in many cases you should decide which approach suits you better in each particular case – it's hard to give the universal advice.
\nClient-side subscriptions are more flexible in general, so I'd suggest using them whenever possible. Though you may use unidirectional transports of Centrifugo where subscribing to channels from the client side is not simple to achieve (though still possible using our server subscribe API). Server-side subscriptions make more sense there.
\nThe good news is that all our official bidirectional client SDKs support all the approaches mentioned in this post. Hope designing the channel configuration on top of Centrifugo will be a pleasant experience for you.
\nToday we are excited to announce the next generation of Centrifugo – Centrifugo v4. The release takes Centrifugo to the next level in terms of client protocol performance, WebSocket fallback simplicity, SDK ecosystem and channel security model. It also comes with a couple of cutting-edge technologies to experiment with such as HTTP/3 and WebTransport.
\nIf you've never heard of Centrifugo before, it's an open-source scalable real-time messaging server written in Go language. Centrifugo can instantly deliver messages to application online users connected over supported transports (WebSocket, HTTP-streaming, Server-Sent Events (SSE), GRPC, SockJS). Centrifugo has the concept of a channel – so it's a user-facing PUB/SUB server.
Centrifugo is language-agnostic and can be used to build chat apps, live comments, multiplayer games, real-time data visualizations, collaborative tools, etc. in combination with any backend. It is well suited for modern architectures and allows decoupling the business logic from the real-time transport layer.
Several official client SDKs for browser and mobile development wrap the bidirectional protocol. In addition, Centrifugo supports a unidirectional approach for simple use cases with no SDK dependency.
Let's start from looking back a bit. Centrifugo v3 was released last year. It had a great list of improvements – like unidirectional transports support (EventSource, HTTP-streaming and GRPC), GRPC transport for proxy, history iteration API, faster JSON protocol, super-fast but experimental Tarantool engine implementation, and others.
\nDuring the Centrifugo v3 lifecycle we added even more JSON protocol optimizations and introduced a granular proxy mode. Experimental Tarantool engine has also evolved a bit.
\nBut Centrifugo v3 did not contain anything... let's say revolutional. Revolutional for Centrifugo itself, community, or even the entire field of open-source real-time messaging.
\nWith this release, we feel that we bring innovation to the ecosystem. Now let's talk about it and introduce all the major things of the brand new v4 release.
\n\nThe most challenging part of Centrifugo project is not a server itself. Client SDKs are the hardest part of the ecosystem. We try to time additional improvements to the SDKs with each major release of the server. But this time the SDKs are the centerpiece of the v4 release.
\nCentrifugo uses bidirectional asynchronous protocol between client and server. On top of this protocol SDK provides a request-response over an asynchronous connection, reconnection logic, subscription management and multiplexing, timeout and error handling, ping-pong, token refresh, etc. Some of these things are not that trivial to implement. And all this should be implemented in different programming languages. As you may know, we have official real-time SDKs in Javascript, Dart, Swift, Java and Go.
\nWhile implementing the same protocol and same functions, all SDKs behaved slightly differently. That was the result of the missing SDK specification. Without a strict SDK spec, it was hard to document things, hard to explain the exact details of the real-time SDK behavior. What we did earlier in the Centrifugo documentation – was pointing users to specific SDK Github repo to look for behaviour details.
\nThe coolest thing about Centrifugo v4 is the next generation SDK API. We now have a client SDK API specification. It's a source of truth for SDKs behavior which try to follow the spec closely.
\nThe new SDK API is the result of several iterations and reflections on possible states, transitions, token refresh mechanism, etc. Users in our Telegram group may remember how it all started:
\n![\"Centrifugo](\"https://centrifugal.dev/assets/images/states_prototype-104a12d8315c3ad7dfd6fc983979e134.jpg\")
And after several iterations these prototypes turned into working mechanisms with well-defined behaviour:
\n![\"Centrifugo](\"https://centrifugal.dev/assets/images/client_state-34264b7a7eee2792baa58bb5bb525d46.png\")
A few things that have been revised from the ground up:
\nWe now also have a separation between temporary and non-temporary protocol errors – this allows us to handle subscription internal server errors on the SDK level, making subscriptions more resilient, with automatic resubscriptions, and to ensure individual subscription failures do not affect the entire connection.
\nThe mechanics described in the client SDK API specification are now implemented in all of our official SDKs. The SDKs now support all major client protocol features that currently exist. We believe this is a big step forward for the Centrifugo ecosystem and community.
\nWebSocket is supported almost everywhere these days. But there is a case that we believe is the last one preventing users to connect over WebSocket - corporate proxies. With the root certificate installed on employee computer machines, these proxies can block WebSocket traffic, even if it's wrapped in a TLS layer. That's really annoying, and often developers choose to not support clients connecting from such \"broken\" environments at all.
\nPrior to v4, Centrifugo users could use the SockJS polyfill library to fill this gap.
\nSockJS is great software – stable and field proven. It is still used by some huge real-time messaging players out there to polyfill the WebSocket transport.
\nBut SockJS is an extra frontend dependency with a bunch of legacy transports, and the future of it is unknown.
\nSockJS comes with a notable overhead – it's an aditional protocol wrapper, consumes more memory per connection on a server (at least when using SockJS-Go library – the only choice for implementing SockJS server in Go language these days). When using SockJS, Centrifugo users were losing the ability to use our main pure WebSocket transport because SockJS uses its own WebSocket implementation on a server side.
\nSockJS does not support binary data transfer – only JSON format can be used with it. As you know, our main WebSocket transport works fine with binary in case of using Protobuf protocol format. So with SockJS we don't have fallback for WebSocket with a binary data transfer.
\nAnd finally, if you want to use SockJS with a distributed backend, you must enable sticky session support on the load-balancer level. This way you can point requests from the client to the server to the correct server node – the one which maintains a persistent unidirectional HTTP connection.
\nWe danced around the idea of replacing SockJS for a long time. But only now we are ready to provide our alternative to it – meet Centrifugo own bidirectional emulation layer. It's based on two additional transports:
\nSo when the fallback is used, you always have a real-time, persistent connection in server -> to -> client direction. Requests in client -> to -> server direction are regular HTTP – similar to how SockJS works. But our bidirectional emulation layer does not require sticky sessions – Centrifugo can proxy client-to-server requests to the correct node in the cluster. Having sticky sessions is an optimization for Centrifugo bidirectional emulation layer, not a requirement. We believe that this is a game changer for our users – no need to bother about proper load balancing, especially since in most cases 95% or even more users will be able to connect using the WebSocket transport.
\nHere is a simplified diagram of how it works:
\n![\"Scheme\"](\"https://centrifugal.dev/assets/images/emulation_scheme-7488282ce671166bd0ca4369bb8af842.png\")
The bidirectional emulation layer is only supported by the Javascript SDK (centrifuge-js) – as we think fallbacks mostly make sense for browsers. If we find use cases where other SDKs can benefit from HTTP based transport – we can expand on them later.
Let's look at example of using this feature from the Javascript side. To use fallbacks, all you need to do is to set up a list of desired transports with endpoints:
\nconst transports = [
{
transport: 'websocket',
endpoint: 'wss://your_centrifugo.com/connection/websocket'
},
{
transport: 'http_stream',
endpoint: 'https://your_centrifugo.com/connection/http_stream'
},
{
transport: 'sse',
endpoint: 'https://your_centrifugo.com/connection/sse'
}
];
const centrifuge = new Centrifuge(transports);
centrifuge.connect()
We are using explicit transport endpoints in the above example due to the fact that transport endpoints can be configured separately in Centrifugo – there is no single entry point for all transports. Like the one in Socket.IO or SockJS when developer can only point client to the base address. In Centrifugo case, we are requesting an explicit transport/endpoint configuration from the SDK user.
By the way, a few advantages of HTTP-based transport over WebSocket:
\nSockJS is still supported by Centrifugo and centrifuge-js, but it's now DEPRECATED.
Not only the API of client SDK has changed, but also the format of Centrifugo protocol messages. New format is more human-readable (in JSON case, of course), has a more compact ping message size (more on that below).
\nThe client protocol is now one-shot encode/decode compatible. Previously, Centrifugo protocol had a layered structure and we had to encode some messages before appending them to the top-level message. Or decode two or three times to unwrap the message envelope. To achieve good performance when encoding and decoding client protocol messages, Centrifugo had to use various optimization techniques – like buffer memory pools, byte slice memory pools.
\nBy restructuring the message format, we were able to avoid layering, which allowed us to slightly increase the performance of encoding/decoding without additional optimization tricks.
\n![\"Scheme\"](\"https://centrifugal.dev/assets/images/avoid_protocol_nesting-fb312e08bc008cbd7d4318c9c8357b44.png\")
We also simplified the client protocol documentation overview a bit.
\nIn many cases in practice (when dealing with persistent connections like WebSocket), pings and pongs are the most dominant types of messages passed between client and server. Your application may have many concurrent connections, but only a few of them receive the useful payload. But at the same time, we still need to send pings and respond with pongs. Thus, optimizing the ping-pong process can significantly reduce server resource usage.
\nOne optimization comes from the revised PING-PONG behaviour. Previous versions of Centrifugo and SDKs sent ping/pong in both \"client->to->server\" and \"server->to->client\" directions (for WebSocket transport). This allowed finding non-active connections on both client and server sides.
\nIn Centrifugo v4 we only send pings from a server to a client and expect pong from a client. On the client-side, we have a timer which fires if there hasn't been a ping from the server within the configured time, so we still have a way to detect closed connections.
\nSending pings only in one direction results in 2 times less ping-pong messages - and this should be really noticable for Centrifugo installations with thousands of concurrent connections. In our experiments with 10k connections, server CPU usage was reduced by 30% compared to Centrifugo v3.
\n![\"Scheme\"](\"https://centrifugal.dev/assets/images/ping_pong_v3_v4-b6f5699b9dc3e68059833185661a1f1f.png\")
Pings and pongs are application-level messages. Ping is just an empty asynchronous reply – for example in JSON case it's a 2-byte message: {}. Pong is an empty command – also, {} in JSON case. Having application-level pings from the server also allows unifying the PING format for all unidirectional transports.
Another improvement is that Centrifugo now randomizes the time it sends first ping to the client (but no longer than the configured ping interval). This allows to spread ping-pongs in time, providing a smoother CPU profile, especially after a massive reconnect scenario.
\nData security and privacy are more important than ever in today's world. And as Centrifugo becomes more popular and widely used, the need to be secure by default only increases.
Previously, by default, clients could subcribe to all channels in a namespace (except private channels, which are now revised – see details below). It was possible to use \"protected\": true option to make namespace protected, but we are not sure if everyone did that. This is extra configuration and additional knowledge on how Centrifugo works.
Also, a common confusion we ran into: if server-side subscriptions were dictated by a connection JWT, many users would expect client-side subscriptions to those channels to not work. But without the protected option enabled, this was not the case.
In Centrifugo v4, by default, it is not possible to subscribe to a channel in a namespace. The namespace must be configured to allow subscriptions from clients, or token authorization must be used. There are a bunch of new namespace options to tune the namespace behavior. Also the ability to provide a regular expression for channels in the namespace.
\nThe new permission-related channel option names better reflect the purpose of the option. For example, compare \"publish\": true and \"allow_publish_for_client\": true. The second one is more readable and provides a better understanding of the effect once turned on.
Centrifugo is now more strict when checking channel name. Only ASCII symbols allowed – it was already mentioned in docs before, but wasn't actually enforced. Now we are fixing this.
\nWe understand that these changes will make running Centrifugo more of a challenge, especially when all you want is a public access to all the channels without worrying too much about permissions. It's still possible to achieve, but now the intent must be expicitly expressed in the config.
\nCheck out the updated documentation about channels and namespaces. Our v4 migration guide contains an automatic converter for channel namespace options.
\nA private channel is a special channel starting with $ that could not be subscribed to without a subscription JWT. Prior to v4, having a known prefix allowed us to distinguish between public channels and private channels. But since namespaces are now non-public by default, this distinction is not really important.
This means 2 things:
\n$)$ can only be subscribed with a subscription JWT, even if they belong to a namespace where subscriptions allowed for all clients. This is for security compatibility between v3 and v4.Another notable change in a subscription JWT – client claim is now DEPRECATED. There is no need to put it in the subscription token anymore. Centrifugo supports it only for backwards compatibility, but it will be completely removed in the future releases.
The reason we're removing client claim is actually interesting. Due to the fact that client claim was a required part of the subscription JWT applications could run into a situation where during the massive reconnect scenario (say, million connections reconnect) many requests for new subscription tokens can be generated because the subscription token must contain the client ID generated by Centrifugo for the new connection. That could make it unusually hard for the application backend to handle the load. With a connection JWT we had no such problem – as connections could simply reuse the previous token to reconnect to Centrifugo.
Now the subscription token behaves just like the connection token, so we get a scalable solution for token-based subscriptions as well.
\nWhat's more, this change paved the way for another big improvement...
\nThe improvement we just mentioned is called optimistic subscriptions. If any of you are familiar with the QUIC protocol, then optimistic subscriptions are somewhat similar to the 0-RTT feature in QUIC. The idea is simple – we can include subscription commands to the first frame sent to the server.
\nPreviously, we sent subscriptions only after receiving a successful Connect Reply to a Connect Command from a server. But with the new changes in token behaviour, it seems so logical to put subscribe commands within the initial connect frame. Especially since Centrifugo protocol always supported batching of commands. Even token-based subscriptions can now be included into the initial frame during reconnect process, since the previous token can be reused now.
\n![](\"https://centrifugal.dev/assets/images/optimistic_subs-8a424a694e135797511598c93f3fb23b.png\")
The benefit is awesome – in most scenarios, we save one RTT of latency when connecting to Centrifugo and subscribing to channels (which is actually the most common way to use Centrifugo). While not visible on localhost, this is pretty important in real-life. And this is less syscalls for the server after all, resulting in less CPU usage.
\nOptimistic subscriptions are also great for bidirectional emulation with HTTP, as they avoid the long path of proxying a request to the correct Centrifugo node when connecting.
\nOptimistic subscriptions are now only part of centrifuge-js. At some point, we plan to roll out this important optimization to all other client SDKs.
The channel capabilities feature is introduced as part of Centrifugo PRO. Initially, we aimed to make it a part of the OSS version. But the lack of feedback on this feature made us nervous it's really needed. So adding it to PRO, where we still have room to evaluate the idea, seemed like the safer decision at the moment.
\nCentrifugo allows configuring channel permissions on a per-namespace level. When creating a new real-time feature, it is recommended to create a new namespace for it and configure permissions. But to achieve a better channel permission control within a namespace the Channel capabilities can be used now.
\nThe channel capability feature provides a possibility to set capabilities on an individual connection basis, or an individual channel subscription basis.
\nFor example, in a connection JWT developers can set sth like:
\n{
\"caps\": [
{
\"channels\": [\"news\", \"user_42\"],
\"allow\": [\"sub\"]
}
]
}
And this tells Centrifugo that the connection is able to subscribe on channels news or user_42 using client-side subscriptionsat any time while the connection is active. Centrifugo also supports wildcard and regex channel matches.
Subscription JWT can provide capabilities for the channel too, so permissions may be controlled on an individual subscription basis, ex. the ability to publish and call history API may be expressed with allow claim in subscription JWT:
{
\"allow\": [\"pub\", \"hst\"]
}
Read more about this mechanism in Channel capabilities chapter.
\nAnother addition to Centrifugo PRO is the improved connection API. Previously, we could only return all connections from a specific user.
\nThe API now supports filtering all connections: by user ID, by subscribed channel, by additional meta information attached to the connection.
\nThe filtering works by user ID or with a help of CEL expressions (Common Expression Language). CEL expressions provide a developer-friendly, fast and secure (as they are not Turing-complete) way to evaluate some conditions. They are used in some Google services (ex. Firebase), in Envoy RBAC configuration, etc. If you've never seen it before – take a look, cool project. We are also evaluating how to use CEL expressions for a dynamic and efficient channel permission checks, but that's an early story.
\nThe connections API call result contains more useful information: a list of client's active channels, information about the tokens used to connect and subscribe, meta information attached to the connection.
It's no secret that centrifuge-js is the most popular SDK in the Centrifugo ecosystem. We put additional love to it – and centrifuge-js is now fully written in Typescript ❤️
This was a long awaited improvement, and it finally happened! The entire public API is strictly typed. The cool thing is that even EventEmitter events and event handlers are the subject to type checks - this should drastically simplify and speedup development and also help to reduce error possibility.
Centrifugo v4 has an experimental HTTP/3 support. Once TLS is enabled and \"http3\": true option is set all the endpoints on an external port will be served by a HTTP/3 server based on lucas-clemente/quic-go implementation.
It's worth noting that WebSocket will still use HTTP/1.1 for its Upgrade request (there is an interesting IETF draft BTW about Bootstrapping WebSockets with HTTP/3). But HTTP-streaming and EventSource should work just fine with HTTP/3.
\nHTTP/3 does not currently work with our ACME autocert TLS - i.e. you need to explicitly provide paths to cert and key files as described here.
\nHaving HTTP/3 on board allowed us to make one more thing. Some of you may remember the post Experimenting with QUIC and WebTransport published in our blog before. We danced around the idea to add WebTransport to Centrifugo since then. WebTransport IETF specification is still a draft, it changed a lot since our first blog post about it. But WebTransport object is already part of Chrome (since v97) and things seem to be very close to the release.
\nSo we added experimental WebTransport support to Centrifugo v4. This is made possible with the help of the marten-seemann/webtransport-go library.
\nTo use WebTransport you need to run HTTP/3 experimental server and enable WebTransport endpoint with \"webtransport\": true option in the configuration. Then you can connect to that endpoint using centrifuge-js. For example, let's enable WebTransport and use WebSocket as a fallback option:
const transports = [
{
transport: 'webtransport',
endpoint: 'https://your_centrifugo.com/connection/webtransport'
},
{
transport: 'websocket',
endpoint: 'wss://your_centrifugo.com/connection/websocket'
}
];
const centrifuge = new Centrifuge(transports);
centrifuge.connect()
Note, that we are using secure schemes here – https:// and wss://. While in WebSocket case you could opt for non-TLS communication, in HTTP/3 and specifically WebTransport non-TLS communication is simply not supported by the specification.
In Centrifugo case, we utilize the bidirectional reliable stream of WebTransport to pass our protocol between client and server. Both JSON and Protobuf communication formats are supported. There are some issues with the proper passing of the disconnect advice in some cases, otherwise it's fully functional.
\nObviously, due to the limited WebTransport support in browsers at the moment, possible breaking changes in the WebTransport specification we can not recommended it for production usage for now. At some point in the future, it may become a reasonable alternative to WebSocket, now we are more confident that Centrifugo will be able to provide a proper support of it.
\nThe migration guide contains steps to upgrade your Centrifugo from version 3 to version 4. While there are many changes in the v4 release, it should be possible to migrate to Centrifugo v4 without changing the code on the client side at all. And then, after updating the server, gradually update the client-side to the latest version of the stack.
\n![](\"https://centrifugal.dev/assets/images/bg_cat-975c20ed8cf4a65568bc628c638050f9.jpg\")
To sum it up, here are some benefits of Centrifugo v4:
\nclient claim not needed anymore)That's it. We now begin the era of v4 and it is going to be awesome, no doubt.
\nThe release contains many changes that strongly affect developing with Centrifugo. And of course you may have some questions or issues regarding new or changed concepts. Join our communities in Telegram (the most active) and Discord:
\n
Enjoy Centrifugo v4, and let the Centrifugal force be with you.
\nThe refactoring of client SDKs and introducing unified behavior based on the common spec was the hardest part of Centrifugo v4 release. Many thanks to Vitaly Puzrin (who is the author of several popular open-source libraries such as markdown-it, fontello, and others). We had a series of super productive sessions with him on client SDK API design. Some great ideas emerged from these sessions and the result seems like a huge step forward for Centrifugal projects.
\nAlso, thanks to Anton Silischev who helped a lot with WebTransport prototypes earlier this year, so we could quickly adopt WebTransport for v4.
\nAs some of you know, Centrifugo server is built on top of the Centrifuge library for Go. Most of the optimizations and improvements described here are now also part of Centrifuge library.
With its new unified SDK behavior and bidirectional emulation layer, it seems a solid alternative to Socket.IO in the Go language ecosystem.
In some cases, Centrifuge library can be a more flexible solution than Centrifugo, since Centrifugo (as a standalone server) dictates some mechanics and rules that must be followed. In the case of Centrifugo, the business logic must live on the application backend side, with Centrifuge library it can be kept closer to the real-time transport layer.
This post used images from freepik.com: background by liuzishan. Also image by kenshinstock.
In this tutorial, we will create a multi-room chat server using Laravel framework and Centrifugo real-time messaging server.
\nAuthenticated users of our chat app will be able to create new chat rooms, join existing rooms and instantly communicate inside rooms with the help of Centrifugo WebSocket real-time transport.
\nThis tutorial was written for Centrifugo v3. We recently released Centrifugo v4 which makes some parts of this tutorial obsolete. The core concepts are similar though – so this can still be used as a Centrifugo learning step.
The result will look like this:
\n\nFor the backend, we are using Laravel (version 8.65) as one of the most popular PHP frameworks. Centrifugo v3 will accept WebSocket client connections. And we will implement an integration layer between Laravel and Centrifugo.
\nFor CSS styles we are using recently released Bootstrap 5. Also, some vanilla JS instead of frameworks like React/Vue/whatever to make frontend Javascript code simple – so most developers out there could understand the mechanics.
\nWe are also using a bit old-fashioned server rendering here where server renders templates for different room routes (URLs) – i.e. our app is not a SPA app – mostly for the same reasons: to keep example short and let reader focus on Centrifugo and Laravel integration parts.
\nTo generate fake user avatars we are requesting images from https://robohash.org/ which can generate unique robot puctures based on some input string (username in our case). Robots like to chat with each other!
\n![](\"https://robohash.org/1.png\")
\n![](\"https://robohash.org/2.png\")
\n![](\"https://robohash.org/4.png\")
\nWe also have some ideas on further possible app improvements at the end of this post.
Why would Laravel developers want to integrate a project with Centrifugo for real-time messaging functionality? That's a good question. There are several points which could be a good motivation:
\nHope this makes sense as a good motivation to give Centrifugo a try in your Laravel project. Let's get started!
\nFor the convenience of working with the example, we wrapped the end result into docker compose.
\nTo start the app clone examples repo, cd into v3/php_laravel_chat_tutorial directory and run:
docker compose up
At the first launch, the necessary images will be downloaded (will take some time and network bytes). When the main service is started, you should see something like this in container logs:
\n...
app | Database seeding completed successfully.
app | [10-Dec-2021 12:25:05] NOTICE: fpm is running, pid 112
app | [10-Dec-2021 12:25:05] NOTICE: ready to handle connections
Then go to http://localhost/ – you should see:
\n![\"Image\"](\"https://centrifugal.dev/assets/images/laravel_main_page-c3e70ae9857eefd3ca4fcd9999a7962c.jpg\")
Register (using some fake credentials) or sign up – and proceed to the chat rooms.
\nPay attention to the configuration of Centrifugo and Nginx. Also, on entrypoint which does some things:
\nWe assume you already familar with Laravel concepts, so we will just point you to some core aspects of the Laravel application structure and will pay more attention to Centrifugo integration parts.
\nAfter the first launch of the application, all settings will be copied from the file .env.example to .env. Next, we will take a closer look at some settings.
You can view the database structure here.
\nWe will use the following tables which will be then translated to the application models:
\njoin button clicked or automatically upon room creation.For broadcasting we are using laravel-centrifugo library. It helps to simplify interaction between Laravel and Centrifugo by providing some convenient wrappers.
\nStep-by-step configuration can be viewed in the readme file of this library.
\nPay attention to the CENTRIFUGO_API_KEY setting. It is used to send API requests from Laravel to Centrifugo and must match in .env and centrifugo.json files. And we also telling laravel-centrifugo the URL of Centrifugo. That's all we need to configure for this example app.
See more information about Laravel broadcasting here.
\nAs an alternative to laravel-centrifugo, you can use phpcent – it's an official generic API client which allows publishing to Centrifugo HTTP API. But it does know nothing about Laravel broadcasting specifics.
When user opens a chat app it connects to Centrifugo over WebSocket transport.
\nLet's take a closer look at Centrifugo server configuration file we use for this example app:
\n{
\"port\": 8000,
\"engine\": \"memory\",
\"api_key\": \"some-long-api-key-which-you-should-keep-secret\",
\"allowed_origins\": [
\"http://localhost\",
],
\"proxy_connect_endpoint\": \"http://nginx/centrifugo/connect/\",
\"proxy_http_headers\": [
\"Cookie\"
],
\"namespaces\": [
{
\"name\": \"personal\"
}
]
}
This configuration defines a connect proxy endpoint which is targeting Nginx and then proxied to Laravel. Centrifugo will proxy Cookie header of WebSocket HTTP Upgrade requests to Laravel – this allows using native Laravel authentication.
We also defined a \"personal\" namespace – we will subscribe each user to a personal channel in this namespace inside connect proxy handler. Using namespaces for different real-time features is one of Centrifugo best-practices.
Allowed origins must be properly set to prevent cross-site WebSocket connection hijacking.
\nTo use native Laravel user authentication middlewares, we will use Centrifugo proxy feature.
\nWhen user connects to Centrifugo it's connection attempt will be transformed into HTTP request from Centrifugo to Laravel and will hit the connect proxy controller:
\nclass CentrifugoProxyController extends Controller
{
public function connect()
{
return new JsonResponse([
'result' => [
'user' => (string) Auth::user()->id,
'channels' => [\"personal:#\".Auth::user()->id],
]
]);
}
}
This controller protected by auth middleware.
\nSince Centrifugo proxies Cookie header of initial WebSocket HTTP Upgrade request Laravel auth layer will work just fine. So in a controller you already has access to the current authenticated user.
In the response from controller we tell Centrifugo the ID of connecting user and subscribe user to its personal channel (using user-limited channel feature of Centrifugo). Returning a channel in such way will subscribe user to it using server-side subscriptions mechanism.
\nNote, that in our chat app we are using a single personal channel for each user to receive real-time updates from all rooms. We are not creating separate subscriptions for each room user joined too. This will allow us to scale more easily in the future, and basically the only viable solution in case of room list pagination in chat application like this. It does not mean you can not combine personal user channels and separate room channels for different tasks though.
Some additional tips can be found in Centrifugo FAQ.
In RoomController we perform various actions with rooms:
\nWhen we publish a message in a room, we send a message to the personal channel of all users joined to the room using the broadcast method of Centrifugo API. It allows publishing the same message into many channels.
$message = Message::create([
'sender_id' => Auth::user()->id,
'message' => $requestData[\"message\"],
'room_id' => $id,
]);
$room = Room::with('users')->find($id);
$channels = [];
foreach ($room->users as $user) {
$channels[] = \"personal:#\" . $user->id;
}
$this->centrifugo->broadcast($channels, [
\"text\" => $message->message,
\"createdAt\" => $message->created_at->toDateTimeString(),
\"roomId\" => $id,
\"senderId\" => Auth::user()->id,
\"senderName\" => Auth::user()->name,
]);
We also add some fields to the published message which will be used when dynamically displaying a message coming from a WebSocket connection (see Client side below).
\nOur chat is basically a one page with some variations dependng on the current route. So we use a single view for the entire chat app.
\nOn the page we have a form for creating rooms. The user who created the room automatically joins it upon creation. Other users need to join manually (using join button in the room).
When sending a message (using the chat room message input), we make an AJAX request that hits RoomController shown above. A message saved into the database and then broadcasted to all users who joined this room. Here is a code that processes sending on ENTER:
messageInput.onkeyup = function(e) {
if (e.keyCode === 13) {
e.preventDefault();
const message = messageInput.value;
if (!message) {
return;
}
const xhttp = new XMLHttpRequest();
xhttp.open(\"POST\", \"/rooms/\" + roomId + \"/publish\");
xhttp.setRequestHeader(\"X-CSRF-TOKEN\", csrfToken);
xhttp.send(JSON.stringify({
message: message
}));
messageInput.value = '';
}
};
After the message is processed on the server and broadcasted to Centrifugo it instantly comes to client-side. To receive the message we are connecting to Centrifugo WebSocket endpoint and wait for a message in the publish event handler:
const url = \"ws://\" + window.location.host + \"/connection/websocket\";
const centrifuge = new Centrifuge(url);
centrifuge.on('connect', function(ctx) {
console.log(\"connected to Centrifugo\", ctx);
});
centrifuge.on('disconnect', function(ctx) {
console.log(\"disconnected from Centrifugo\", ctx);
});
centrifuge.on('publish', function(ctx) {
if (ctx.data.roomId.toString() === currentRoomId) {
addMessage(ctx.data);
scrollToLastMessage();
}
addRoomLastMessage(ctx.data);
});
centrifuge.connect();
We are using centrifuge-js client connector library to communicate with Centrifugo. This client abstracts away bidirectional asynchronous protocol complexity for us providing a simple way to listen connect, disconnect events and communicate with a server in various ways.
\nIn publish event handler we check whether the message belongs to the room the user is currently in. If yes, then we add it to the message history of the room. We also add this message to the room in the list on the left as the last chat message in room. If necessary, we crop the text for normal display.
\nIn our example we only subscribe each user to a single channel, but user can be subscribed to several server-side channels. To distinguish between them use ctx.channel inside publish event handler.
And that's it! We went through all the main parts of the integration.
\nAs promised, here is a list with several possible app improvements:
\nWe built a chat app with Laravel and Centrifugo. While there is still an area for improvements, this example is not really the basic. It's already valuable in the current form and may be transformed into part of your production system with minimal tweaks.
\nHope you enjoyed this tutorial. If you have any questions after reading – join our community channels. We touched only part of Centrifugo concepts here – take a look at detailed Centrifugo docs nearby. And let the Centrifugal force be with you!
", "url": "https://centrifugal.dev/blog/2021/12/14/laravel-multi-room-chat-tutorial", "title": "Building a multi-room chat application with Laravel and Centrifugo", "summary": "In this tutorial, we are integrating Laravel framework with Centrifugo real-time messaging server to make a multi-room chat application.", "date_modified": "2021-12-14T00:00:00.000Z", "author": { "name": "Anton Silischev" }, "tags": [ "centrifugo", "tutorial", "laravel", "php" ] }, { "id": "https://centrifugal.dev/blog/2021/11/04/integrating-with-django-building-chat-application", "content_html": "In this tutorial, we will create a basic chat server using the Django framework and Centrifugo. Our chat application will have two pages:
\nThe room view will use a WebSocket to communicate with the Django server (with help from Centrifugo) and listen for any messages that are published to the room channel.
\nThis tutorial was written for Centrifugo v3. We recently released Centrifugo v4 which makes some parts of this tutorial obsolete. The core concepts are similar though – so this can still be used as a Centrifugo learning step.
The result will look like this:
\n![\"demo\"](\"https://centrifugal.dev/assets/images/django_chat-9972126f452363132e76a62f00213fbf.gif\")
Some of you will notice that this tutorial looks very similar to Chat app tutorial of Django Channels. This is intentional to let Pythonistas already familiar with Django Channels feel how Centrifugo compares to Channels in terms of the integration process.
Why would Django developers want to integrate a project with Centrifugo for real-time messaging functionality? This is a good question especially since there is a popular Django Channels project which solves the same task.
\nI found several points which could be a good motivation:
\nWe assume that you are already familiar with basic Django concepts. If not take a look at the official Django tutorial first and then come back to this tutorial.
\nAlso, make sure you read a bit about Centrifugo – introduction and quickstart tutorial.
\nWe also assume that you have Django installed already.
\nOne possible way to quickly install Django locally is to create virtualenv, activate it, and install Django:
\npython3 -m venv env
. env/bin/activate
pip install django
Alos, make sure you have Centrifugo v3 installed already.
\nThis tutorial also uses Docker to run Redis. We use Redis as a Centrifugo engine – this allows us to have a scalable solution in the end. Using Redis is optional actually, Centrifugo uses a Memory engine by default (but it does not allow scaling Centrifugo nodes). We will also run Nginx with Docker to serve the entire app. Install Docker from its official website but I am sure you already have one.
\nFirst, let's create a Django project.
\nFrom the command line, cd into a directory where you’d like to store your code, then run the following command:
django-admin startproject mysite
This will create a mysite directory in your current directory with the following contents:
\n❯ tree mysite
mysite
├── manage.py
└── mysite
├── __init__.py
├── asgi.py
├── settings.py
├── urls.py
└── wsgi.py
We will put the code for the chat server inside chat app.
Make sure you’re in the same directory as manage.py and type this command:
python3 manage.py startapp chat
That’ll create a directory chat, which is laid out like this:
\n❯ tree chat
chat
├── __init__.py
├── admin.py
├── apps.py
├── migrations
│ └── __init__.py
├── models.py
├── tests.py
└── views.py
For this tutorial, we will only be working with chat/views.py and chat/__init__.py. Feel free to remove all other files from the chat directory.
After removing unnecessary files, the chat directory should look like this:
\n❯ tree chat
chat
├── __init__.py
└── views.py
We need to tell our project that the chat app is installed. Edit the mysite/settings.py file and add 'chat' to the INSTALLED_APPS setting. It’ll look like this:
# mysite/settings.py
INSTALLED_APPS = [
'chat',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
]
We will now create the first view, an index view that lets you type the name of a chat room to join.
\nCreate a templates directory in your chat directory. Within the templates directory, you have just created, create another directory called chat, and within that create a file called index.html to hold the template for the index view.
Your chat directory should now look like this:
\n❯ tree chat
chat
├── __init__.py
├── templates
│ └── chat
│ └── index.html
└── views.py
Put the following code in chat/templates/chat/index.html:
\n<!DOCTYPE html>
<html>
<head>
<meta charset=\"utf-8\" />
<title>Select a chat room</title>
</head>
<body>
<div class=\"center\">
<div class=\"input-wrapper\">
<input type=\"text\" id=\"room-name-input\" />
</div>
<div class=\"input-help\">
Type a room name to <a id=\"room-name-submit\" href=\"#\">JOIN</a>
</div>
</div>
<script>
const nameInput = document.querySelector('#room-name-input');
const nameSubmit = document.querySelector('#room-name-submit');
nameInput.focus();
nameInput.onkeyup = function (e) {
if (e.keyCode === 13) { // enter, return
nameSubmit.click();
}
};
nameSubmit.onclick = function (e) {
e.preventDefault();
var roomName = nameInput.value;
if (!roomName) {
return;
}
window.location.pathname = '/chat/room/' + roomName + '/';
};
</script>
</body>
</html>
Create the view function for the room view. Put the following code in chat/views.py:
from django.shortcuts import render
def index(request):
return render(request, 'chat/index.html')
To call the view, we need to map it to a URL - and for this, we need a URLconf.
\nTo create a URLconf in the chat directory, create a file called urls.py. Your app directory should now look like this:
❯ tree chat
chat
├── __init__.py
├── templates
│ └── chat
│ └── index.html
└── views.py
└── urls.py
In the chat/urls.py file include the following code:
from django.urls import path
from . import views
urlpatterns = [
path('', views.index, name='index'),
]
The next step is to point the root URLconf at the chat.urls module. In mysite/urls.py, add an import for django.conf.urls.include and insert an include() in the urlpatterns list, so you have:
from django.conf.urls import include
from django.urls import path
from django.contrib import admin
urlpatterns = [
path('chat/', include('chat.urls')),
path('admin/', admin.site.urls),
]
Let’s verify that the index view works. Run the following command:
\npython3 manage.py runserver
You’ll see the following output on the command line:
\nWatching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
You have 18 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
October 21, 2020 - 18:49:39
Django version 3.1.2, using settings 'mysite.settings'
Starting development server at http://localhost:8000/
Quit the server with CONTROL-C.
Go to http://localhost:8000/chat/ in your browser and you should see the a text input to provide a room name.
\nType in \"lobby\" as the room name and press Enter. You should be redirected to the room view at http://localhost:8000/chat/room/lobby/ but we haven’t written the room view yet, so you’ll get a \"Page not found\" error page.
\nGo to the terminal where you ran the runserver command and press Control-C to stop the server.
\nWe will now create the second view, a room view that lets you see messages posted in a particular chat room.
\nCreate a new file chat/templates/chat/room.html. Your app directory should now look like this:
chat
├── __init__.py
├── templates
│ └── chat
│ ├── index.html
│ └── room.html
├── urls.py
└── views.py
Create the view template for the room view in chat/templates/chat/room.html:
<!DOCTYPE html>
<html>
<head>
<meta charset=\"utf-8\" />
<title>Chat Room</title>
<script src=\"https://cdn.jsdelivr.net/gh/centrifugal/centrifuge-js@2.8.3/dist/centrifuge.min.js\"></script>
</head>
<body>
<ul id=\"chat-thread\" class=\"chat-thread\"></ul>
<div class=\"chat-message\">
<input id=\"chat-message-input\" class=\"chat-message-input\" type=\"text\" autocomplete=\"off\" autofocus />
</div>
{{ room_name|json_script:\"room-name\" }}
<script>
const roomName = JSON.parse(document.getElementById('room-name').textContent);
const chatThread = document.querySelector('#chat-thread');
const messageInput = document.querySelector('#chat-message-input');
const centrifuge = new Centrifuge(\"ws://\" + window.location.host + \"/connection/websocket\");
centrifuge.on('connect', function (ctx) {
console.log(\"connected\", ctx);
});
centrifuge.on('disconnect', function (ctx) {
console.log(\"disconnected\", ctx);
});
const sub = centrifuge.subscribe('rooms:' + roomName, function (ctx) {
const chatNewThread = document.createElement('li');
const chatNewMessage = document.createTextNode(ctx.data.message);
chatNewThread.appendChild(chatNewMessage);
chatThread.appendChild(chatNewThread);
chatThread.scrollTop = chatThread.scrollHeight;
});
centrifuge.connect();
messageInput.focus();
messageInput.onkeyup = function (e) {
if (e.keyCode === 13) { // enter, return
e.preventDefault();
const message = messageInput.value;
if (!message) {
return;
}
sub.publish({ 'message': message });
messageInput.value = '';
}
};
</script>
</body>
</html>
Create the view function for the room view in chat/views.py:
from django.shortcuts import render
def index(request):
return render(request, 'chat/index.html')
def room(request, room_name):
return render(request, 'chat/room.html', {
'room_name': room_name
})
Create the route for the room view in chat/urls.py:
# chat/urls.py
from django.urls import path, re_path
from . import views
urlpatterns = [
path('', views.index, name='index'),
re_path('room/(?P<room_name>[A-z0-9_-]+)/', views.room, name='room'),
]
Start the development server:
\npython3 manage.py runserver
Go to http://localhost:8000/chat/ in your browser and to see the index page.
\nType in \"lobby\" as the room name and press enter. You should be redirected to the room page at http://localhost:8000/chat/lobby/ which now displays an empty chat log.
\nType the message \"hello\" and press Enter. Nothing happens! In particular, the message does not appear in the chat log. Why?
\nThe room view is trying to open a WebSocket connection with Centrifugo using the URL ws://localhost:8000/connection/websocket but we haven’t started Centrifugo to accept WebSocket connections yet. If you open your browser’s JavaScript console, you should see an error that looks like this:
WebSocket connection to 'ws://localhost:8000/connection/websocket' failed
And since port 8000 has already been allocated we will start Centrifugo at a different port actually.
\nAs promised we will use Centrifugo with Redis engine. So first thing to do before running Centrifugo is to start Redis:
\ndocker run -it --rm -p 6379:6379 redis:6
Then create a configuration file for Centrifugo:
\n{
\"port\": 8001,
\"engine\": \"redis\",
\"redis_address\": \"redis://localhost:6379\",
\"allowed_origins\": \"http://localhost:9000\",
\"proxy_connect_endpoint\": \"http://localhost:8000/chat/centrifugo/connect/\",
\"proxy_publish_endpoint\": \"http://localhost:8000/chat/centrifugo/publish/\",
\"proxy_subscribe_endpoint\": \"http://localhost:8000/chat/centrifugo/subscribe/\",
\"proxy_http_headers\": [\"Cookie\"],
\"namespaces\": [
{
\"name\": \"rooms\",
\"publish\": true,
\"proxy_publish\": true,
\"proxy_subscribe\": true
}
]
}
And run Centrifugo with it like this:
\ncentrifugo -c config.json
Let's describe some options we used here:
\nport - sets the port Centrifugo runs on since we are running everything on localhost we make it different (8001) from the port allocated for the Django server (8000).engine - as promised we are using Redis engine so we can easily scale Centrifigo nodes to handle lots of WebSocket connectionsredis_address allows setting Redis addressallowed_origins - we will connect from http://localhost:9000 so we need to allow itnamespaces – we are using rooms: prefix when subscribing to a channel, i.e. using Centrifugo rooms namespace. Here we define this namespace and tell Centrifigo to proxy subscribe and publish events for channels in the namespace.It's a good practice to use different namespaces in Centrifugo for different real-time features as this allows enabling only required options for a specific task.
Also, config has some options related to Centrifugo proxy feature. This feature allows proxying WebSocket events to the configured endpoints. We will proxy three types of events:
\nIn Centrifugo config we set endpoints which we will soon implement inside our Django app. You may notice that the allowed origin has a URL with port 9000. That's because we want to proxy Cookie headers from a persistent connection established with Centrifugo to the Django app and need Centrifugo and Django to share the same origin (so browsers can send Django session cookies to Centrifugo).
While not used in this tutorial (we will use fake tutorial-user as user ID here) – this can be useful if you decide to authenticate connections using Django native sessions framework later. To achieve this we should also add Nginx with a configuration like this:
events {
worker_connections 1024;
}
error_log /dev/stdout info;
http {
access_log /dev/stdout;
server {
listen 9000;
server_name localhost;
location / {
proxy_pass http://host.docker.internal:8000;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location /connection/websocket {
proxy_pass http://host.docker.internal:8001;
proxy_http_version 1.1;
proxy_buffering off;
keepalive_timeout 65;
proxy_read_timeout 60s;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_cache_bypass $http_upgrade;
}
}
}
Start Nginx (replace the path to nginx.conf to yours):
docker run -it --rm -v /path/to/nginx.conf:/etc/nginx/nginx.conf:ro -p 9000:9000 --add-host=host.docker.internal:host-gateway nginx
Note that we are exposing port 9000 to localhost and use a possibility to use host.docker.internal host to communicate from inside Docker network with services which are running on localhost (on the host machine). See this answer on SO.
Open http://localhost:9000. Nginx should now properly proxy requests to Django server and to Centrifugo, but we still need to do some things.
\nWell, now if you try to open a chat page with Nginx, Centrifugo, Django, and Redis running you will notice some errors in Centrifugo logs. That's because Centrifugo tries to proxy WebSocket connect events to Django to authenticate them but we have not created event handlers in Django yet. Let's fix this.
\nExtend chat/urls.py:
\nfrom django.urls import path, re_path
from . import views
urlpatterns = [
path('', views.index, name='index'),
re_path('room/(?P<room_name>[A-z0-9_-]+)/', views.room, name='room'),
path('centrifugo/connect/', views.connect, name='connect'),
path('centrifugo/subscribe/', views.subscribe, name='subscribe'),
path('centrifugo/publish/', views.publish, name='publish'),
]
Extend chat/views.py:
\nfrom django.http import JsonResponse
from django.views.decorators.csrf import csrf_exempt
@csrf_exempt
def connect(request):
# In connect handler we must authenticate connection.
# Here we return a fake user ID to Centrifugo to keep tutorial short.
# More details about connect result format can be found in proxy docs:
# https://centrifugal.dev/docs/server/proxy#connect-proxy
logger.debug(request.body)
response = {
'result': {
'user': 'tutorial-user'
}
}
return JsonResponse(response)
@csrf_exempt
def publish(request):
# In publish handler we can validate publication request initialted by a user.
# Here we return an empty object – thus allowing publication.
# More details about publish result format can be found in proxy docs:
# https://centrifugal.dev/docs/server/proxy#publish-proxy
response = {
'result': {}
}
return JsonResponse(response)
@csrf_exempt
def subscribe(request):
# In subscribe handler we can validate user subscription request to a channel.
# Here we return an empty object – thus allowing subscription.
# More details about subscribe result format can be found in proxy docs:
# https://centrifugal.dev/docs/server/proxy#subscribe-proxy
response = {
'result': {}
}
return JsonResponse(response)
connect view will accept all connections and return user ID as tutorial-user. In real app you most probably want to use Django sessions and return real authenticated user ID instead of tutorial-user. Since we told Centrifugo to proxy connection Cookie headers native Django user authentication will work just fine.
Restart Django and try the chat app again. You should now successfully connect. Open a browser tab to the room page at http://localhost:9000/chat/room/lobby/. Open a second browser tab to the same room page.
\nIn the second browser tab, type the message \"hello\" and press Enter. You should now see \"hello\" echoed in the chat log in both the second browser tab and in the first browser tab.
\nYou now have a basic fully-functional chat server!
\nThe list is large, but it's fun to do. To name some possible improvements:
\ntutorial-user used here with native Django session framework. We already proxying the Cookie header to Django from Centrifugo, so you can reuse native Django authentication. Only allow authenticated users to join rooms.Room model and add users to it – thus you will be able to check permissions inside subscribe and publish handlers.Message model to display chat history in Room.One more thing I'd like to note is that if you aim to build a chat application like WhatsApp or Telegram where you have a screen with list of chats (which can be pretty long!) you should not create a separate channel for each room. In this case using separate channel per room does not scale well and you better use personal channel for each user to receive all user-related messages. And as soon as message published to a chat you can send message to each participant's channel. In this case, take a look at Centrifugo broadcast API.
\nThe full example which can run by issuing a single docker compose up can be found on Github. It also has some CSS styles so that the chat looks like shown in the beginning.
Here we implemented a basic chat app with Django and Centrifugo.
\nWhile a chat still requires work to be suitable for production this example can help understand core concepts of Centrifugo - specifically channel namespaces and proxy features.
\nIt's possible to use unidirectional Centrifugo transports instead of bidirectional WebSocket used here – in this case, you can go without using centrifuge-js at all.
Centrifugo scales perfectly if you need to handle more connections – thanks to Centrifugo built-in PUB/SUB engines.
\nIt's also possible to use server-side subscriptions, keep channel history cache, use JWT authentication instead of connect proxy, enable channel presence, and more. All the power of Centrifugo is in your hands.
\nHope you enjoyed this tutorial. And let the Centrifugal force be with you!
\nJoin our community channels in case of any questions left after reading this.
", "url": "https://centrifugal.dev/blog/2021/11/04/integrating-with-django-building-chat-application", "title": "Centrifugo integration with Django – building a basic chat application", "summary": "In this tutorial, we are integrating Django with Centrifugo to make a basic chat application. We are using Centrifugo proxy feature to proxy WebSocket connection events to a Django backend.", "date_modified": "2021-11-04T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "tutorial", "django" ] }, { "id": "https://centrifugal.dev/blog/2021/10/18/integrating-with-nodejs", "content_html": "Centrifugo is a scalable real-time messaging server in a language-agnostic way. In this tutorial we will integrate Centrifugo with NodeJS backend using a connect proxy feature of Centrifugo for user authentication and native session middleware of ExpressJS framework.
\nWhy would NodeJS developers want to integrate a project with Centrifugo? This is a good question especially since there are lots of various tools for real-time messaging available in NodeJS ecosystem.
\nThis tutorial was written for Centrifugo v3. We recently released Centrifugo v4 which makes some parts of this tutorial obsolete. The core concepts are similar though – so this can still be used as a Centrifugo learning step.
I found several points which could be a good motivation:
\nHaving said this all – let's move to a tutorial itself.
\nNot a super-cool app to be honest. Our goal here is to give a reader an idea how integration with Centrifugo could look like. There are many possible apps which could be built on top of this knowledge.
\nThe end result here will allow application user to authenticate and once authenticated – connect to Centrifugo. Centrifugo will proxy connection requests to NodeJS backend and native ExpressJS session middleware will be used for connection authentication. We will also send some periodical real-time messages to a user personal channel.
\nThe full source code of this tutorial located on Github. You can clone examples repo and run this demo by simply writing:
\ndocker compose up
Start new NodeJS app:
\nnpm init
Install dependencies:
\nnpm install express express-session cookie-parser axios morgan
Create index.js file.
const express = require('express');
const cookieParser = require(\"cookie-parser\");
const sessions = require('express-session');
const morgan = require('morgan');
const axios = require('axios');
const app = express();
const port = 3000;
app.use(express.json());
const oneDay = 1000 * 60 * 60 * 24;
app.use(sessions({
secret: \"this_is_my_secret_key\",
saveUninitialized: true,
cookie: { maxAge: oneDay },
resave: false
}));
app.use(cookieParser());
app.use(express.urlencoded({ extended: true }))
app.use(express.json())
app.use(express.static('static'));
app.use(morgan('dev'));
app.get('/', (req, res) => {
if (req.session.userid) {
res.sendFile('views/app.html', { root: __dirname });
} else
res.sendFile('views/login.html', { root: __dirname })
});
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`);
});
Create login.html file in views folder:
<html>
<body>
<form action=\"/login\" method=\"post\">
<h2>Login (username: demo-user, password: demo-pass)</h2>
<div class=\"input-field\">
<input type=\"text\" name=\"username\" id=\"username\" placeholder=\"Enter Username\">
</div>
<div class=\"input-field\">
<input type=\"password\" name=\"password\" id=\"password\" placeholder=\"Enter Password\">
</div>
<input type=\"submit\" value=\"Log in\">
</form>
</body>
</html>
Also create app.html file in views folder:
<html>
<head>
<link rel=\"stylesheet\" href=\"app.css\">
<script src=\"https://cdn.jsdelivr.net/gh/centrifugal/centrifuge-js@2.8.3/dist/centrifuge.min.js\"></script>
</head>
<body>
<div>
<a href='/logout'>Click to logout</a>
</div>
<div id=\"log\"></div>
</body>
</html>
Make attention that we import centrifuge-js client here which abstracts away Centrifugo bidirectional WebSocket protocol.
Let's write an HTTP handler for login form:
\nconst myusername = 'demo-user'
const mypassword = 'demo-pass'
app.post('/login', (req, res) => {
if (req.body.username == myusername && req.body.password == mypassword) {
req.session.userid = req.body.username;
res.redirect('/');
} else {
res.send('Invalid username or password');
}
});
In this example we use hardcoded username and password for out single user. Of course in real app you will have a database with user credentials. But since our goal is only show integration with Centrifugo – we are skipping these hard parts here.
\nAlso create a handler for a logout request:
\napp.get('/logout', (req, res) => {
req.session.destroy();
res.redirect('/');
});
Now if you run an app with node index.js you will see a login form using which you can authenticate. At this point this is a mostly convenient NodeJS application, let's add Centrifugo integration.
Run Centrifugo with config.json like this:
{
\"token_hmac_secret_key\": \"secret\",
\"admin\": true,
\"admin_password\": \"password\",
\"admin_secret\": \"my_admin_secret\",
\"api_key\": \"my_api_key\",
\"allowed_origins\": [
\"http://localhost:9000\"
],
\"user_subscribe_to_personal\": true,
\"proxy_connect_endpoint\": \"http://localhost:3000/centrifugo/connect\",
\"proxy_http_headers\": [
\"Cookie\"
]
}
I.e.:
\n./centrifugo -c config.json
Create app.js file in static folder:
function drawText(text) {
const div = document.createElement('div');
div.innerHTML = text;
document.getElementById('log').appendChild(div);
}
const centrifuge = new Centrifuge('ws://localhost:9000/connection/websocket');
centrifuge.on('connect', function () {
drawText('Connected to Centrifugo');
});
centrifuge.on('disconnect', function () {
drawText('Disconnected from Centrifugo');
});
centrifuge.on('publish', function (ctx) {
drawText('Publication, time = ' + ctx.data.time);
});
centrifuge.connect();
Since we are going to use native session auth of ExpressJS we can't just connect from localhost:3000 (where our NodeJS app is served) to Centrifugo running on localhost:8000 – browser won't send a Cookie header to Centrifugo in this case. Due to this reason we need a reverse proxy which will terminate a traffic from frontend and proxy requests to NodeJS process or to Centrifugo depending on URL path. In this case both browser and NodeJS app will share the same origin – so Cookie will be sent to Centrifugo in WebSocket Upgrade request.
Alternatively, we could also use JWT authentication of Centrifugo but that's a topic for another tutorial. Here we are using connect proxy feature for auth.
Nginx config will look like this:
\nserver {
listen 9000;
server_name localhost;
location / {
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location /connection {
proxy_pass http://localhost:8000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_cache_bypass $http_upgrade;
}
}
Run Nginx and open http://localhost:9000. After authenticating in app you should see an attempt to connect to a WebSocket endpoint. But connection will fail since we need to implement connect proxy handler in NodeJS app.
\napp.post('/centrifugo/connect', (req, res) => {
if (req.session.userid) {
res.json({
result: {
user: req.session.userid
}
});
} else
res.json({
disconnect: {
code: 1000,
reason: \"unauthorized\",
reconnect: false
}
});
});
Restart NodeJS process and try opening an app again. Application should now successfully connect to Centrifugo.
\nLet's also periodically publish current server time to a client's personal channel. In Centrifugo configuration we set a user_subscribe_to_personal option which turns on automatic subscription to a personal channel for each connected user. We can use axios library and send publish API requests to Centrifugo periodically (according to API docs):
const centrifugoApiClient = axios.create({
baseURL: `http://centrifugo:8000/api`,
headers: {
Authorization: `apikey my_api_key`,
'Content-Type': 'application/json',
},
});
setInterval(async () => {
try {
await centrifugoApiClient.post('', {
method: 'publish',
params: {
channel: '#' + myusername, // construct personal channel name.
data: {
time: Math.floor(new Date().getTime() / 1000),
},
},
});
} catch (e) {
console.error(e.message);
}
}, 5000);
After restarting NodeJS you should see periodical updates on application web page.
\nYou can also log in into Centrifugo admin web UI http://localhost:8000 using password password - and play with other available server API from within web interface.
While not being super useful this example can help understanding core concepts of Centrifugo - specifically connect proxy feature and server API.
\nIt's possible to use unidirectional Centrifugo transports instead of bidrectional WebSocket used here – in this case you can go without using centrifuge-js at all.
This application scales perfectly if you need to handle more connections – thanks to Centrifugo builtin PUB/SUB engines.
\nIt's also possible to use client-side subscriptions, keep channel history cache, enable channel presence and more. All the power of Centrifugo is in your hands.
", "url": "https://centrifugal.dev/blog/2021/10/18/integrating-with-nodejs", "title": "Centrifugo integration with NodeJS tutorial", "summary": "In this tutorial we are integrating Centrifugo with NodeJS. We are using Centrifugo connect proxy feature to authenticate connections over standard Express.js session middleware.", "date_modified": "2021-10-18T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifugo", "tutorial", "proxy" ] }, { "id": "https://centrifugal.dev/blog/2021/08/31/hello-centrifugo-v3", "content_html": "After almost three years of Centrifugo v2 life cycle we are happy to announce the next major release of Centrifugo. During the last several months deep in our Centrifugal laboratory we had been synthesizing an improved version of the server.
\nNew Centrifugo v3 is targeting to improve Centrifugo adoption for basic real-time application cases, improves server performance and extends existing features with new functionality. It comes with unidirectional real-time transports, protocol speedups, super-fast engine implementation based on Tarantool, new documentation site, GRPC proxy, API extensions and PRO version which provides unique possibilities for business adopters.
\nCentrifugo v2 life cycle has come to an end. Before discussing v3 let's look back at what has been done during the last three years.
\nCentrifugo v2 was a pretty huge refactoring of v1. Since the v2 release, Centrifugo is built on top of new Centrifuge library for Go language. Centrifuge library evolved significantly since its initial release and now powers Grafana v8 real-time streaming among other things.
\nHere is an awesome demo made by my colleague Alexander Zobnin that demonstrates real-time telemetry of Assetto Corsa sports car streamed in real-time to Grafana dashboard:
\n\n\nCentrifugo integrated with Redis Streams, got Redis Cluster support, can now work with Nats server as a PUB/SUB broker. Notable additions of Centrifugo v2 were server-side subscriptions with some interesting features on top – like maintaining a single global connection from one user and automatic personal channel subscription upon user connect.
\nA very good addition which increased Centrifugo adoption a lot was introduction of proxy to backend. This made Centrifugo fit many setups where JWT authentication and existing subscription permission model did not suit well before.
\nClient ecosystem improved significantly. The fact that client protocol migrated to a strict Protobuf schema allowed to introduce binary protocol format (in addition to JSON) and simplify building client connectors. We now have much better and complete client libraries (compared to v1 situation).
\nWe also have an official Helm chart, Grafana dashboard for Prometheus datasource, and so on.
\nCentrifugo is becoming more noticeable in a wider real-time technology community. For example, it was included in a periodic table of real-time created by Ably.com (one of the most powerful real-time messaging cloud services at the moment):
\n![\"\"](\"https://ik.imagekit.io/ably/ghost/prod/2021/08/periodic-table-screenshots-combined-without-banner-no-legend.jpg?tr=w-1520\")
Of course, there are many aspects where Centrifugo can be improved. And v3 addresses some of them. Below we will look at the most notable features and changes of the new major Centrifugo version.
\nLet's start with the most important thing – backwards compatibility concerns.
\nIn Centrifugo v3 client protocol mostly stayed the same. We expect that most applications will be able to update without any change on a client-side. This was an important concern for v3 given how painful the update cycle can be on mobile devices and lessons learned from v1 to v2 migration. There is one breaking change though which can affect users who use history API manually from a client-side (we provide a temporary workaround to give apps a chance to migrate smoothly).
\nOn a server-side, much more changes happened, especially in the configuration: some options were renamed, some were removed. We provide a v2 to v3 configuration converter which can help dealing with changes. In most cases, all you should do is adapt Centrifugo configuration to match v3 changes and redeploy Centrifugo using v3 build instead of v2. All features are still there (or a replacement exists, like for channels API).
For more details, refer to the v3 migration guide.
\nAs some of you know we considered changing Centrifugo license to AGPL v3 for a new release. After thinking a lot about this we decided to not step into this area.
\nBut the license has been changed: the license of OSS Centrifugo is now Apache 2.0 instead of MIT. Apache 2.0 is also a permissive OSS license, it's just a bit more concrete in some aspects.
\n![\"\"](\"https://user-images.githubusercontent.com/2097922/91162089-8570e100-e6c3-11ea-8c41-cd8fcfe049d0.png\")
Server-side subscriptions introduced in Centrifugo v2 and recent improvements in the underlying Centrifuge library opened a road for a unidirectional approach.
\nThis means that Centrifugo v3 provides a set of unidirectional real-time transports where messages flow only in one direction – from a server to a client. Why is this change important?
\nCentrifugo originally concentrated on using bidirectional transports for client-server communication. Like WebSocket and SockJS. Bidirectional transports allow implementing some great protocol features since a client can communicate with a server in various ways after establishing a persistent connection. While this is a great opportunity this also leads to an increased complexity.
\nCentrifugo users had to use special client connector libraries which abstracted underlying work into a simple public API. But internally connectors do many things: matching requests to responses, handling timeouts, handling an ordering, queuing operations, error handling. So the client connector is a pretty complex piece of software.
\nBut what if a user just needs to receive real-time updates from a stable set of channels known in connection time? Can we simplify everything and avoid using custom software on a client-side?
\nWith unidirectional transports, the answer is yes. Clients can now connect to Centrifugo using a bunch of unidirectional transports. And the greatest thing is that in this case, developers should not depend on Centrifugo client connectors at all – just use native browser APIs or GRPC-generated code. It's finally possible to consume events from Centrifugo using CURL (see an example).
\nUsing unidirectional transports you can still benefit from Centrifugo built-in scalability with various engines, utilize built-in authentication over JWT or the connect proxy feature.
\nWith subscribe server API (see below) it's even possible to subscribe unidirectional client to server-side channels dynamically. With refresh server API or the refresh proxy feature it's possible to manage a connection expiration.
\nCentrifugo supports the following unidirectional transports:
\n\nWe expect that introducing unidirectional transports will significantly increase Centrifugo adoption.
\n![](\"https://centrifugal.dev/img/centrifuge.svg\")
\nThere was a rather important limitation of Centrifugo history API – it was not very suitable for keeping large streams because a call to a history could only return the entire channel history.
\nCentrifugo v3 introduces an API to iterate over a stream. It's possible to do from the current stream beginning or end, in both directions – forward and backward, with configured limit. Also with certain starting stream position if it's known.
\nThis, among other things, can help to implement manual missed message recovery on a client-side to reduce the load on the application backend.
\nHere is an example program in Go which endlessly iterates over stream both ends (using gocent API library), upon reaching the end of stream the iteration goes in reversed direction (not really useful in real world but fun):
\n![\"\"](\"https://grafana.com/api/dashboards/13039/images/8950/image\")
// Iterate by 10.
limit := 10
// Paginate in reversed order first, then invert it.
reverse := true
// Start with nil StreamPosition, then fill it with value while paginating.
var sp *gocent.StreamPosition
for {
\thistoryResult, err = c.History(
ctx,
channel,
\t\tgocent.WithLimit(limit),
\t\tgocent.WithReverse(reverse),
gocent.WithSince(sp),
\t)
\tif err != nil {
\t\tlog.Fatalf(\"Error calling history: %v\", err)
\t}
\tfor _, pub := range historyResult.Publications {
\t\tlog.Println(pub.Offset, \"=>\", string(pub.Data))
\t\tsp = &gocent.StreamPosition{
\t\t\tOffset: pub.Offset,
\t\t\tEpoch: historyResult.Epoch,
\t\t}
\t}
\tif len(historyResult.Publications) < limit {
\t\t// Got all pubs, invert pagination direction.
\t\treverse = !reverse
\t\tlog.Println(\"end of stream reached, change iteration direction\")
\t}
}
This new API does not remove the need in having the main application database – that's still mandatory for idiomatic Centrifugo usage.
In Centrifugo v3 Redis engine uses Redis Stream data structure by default for keeping channel history. Before v3 Redis Streams were supported by not enabled by default so almost nobody used them. This change is important in terms of introducing history iteration API described above – since Redis Streams allow doing iteration effectively.
\nAs you may know, Centrifugo has several built-in engines that allow scaling Centrifugo nodes (using PUB/SUB) and keep shared history and presence state. Before v3 Centrifugo had in-memory and Redis (or KeyDB) engines available.
\nIntroducing a new engine to Centrifugo is pretty hard since the engine should provide a very robust PUB/SUB performance, fast history and presence operations, possibility to publish a message to PUB/SUB and save to history atomically. It also should allow dealing with ephemeral frequently changing subscriptions. It's typical for Centrifugo use case to have millions of users each subscribed to a unique channel and constantly connecting/disconnecting (thus subscribing/unsubscribing).
\n![\"\"](\"https://www.tadviser.ru/images/thumb/1/1a/Tarantool_%D0%A1%D0%A3%D0%91%D0%94_logo_2020.png/840px-Tarantool_%D0%A1%D0%A3%D0%91%D0%94_logo_2020.png\"){kind=logo}
In v3 we added experimental support for the Tarantool engine. It fits nicely all the requirements above and provides a huge performance speedup for history and presence operations compared to Redis. According to our benchmarks, the speedup can be up to 4-10x depending on operation. The PUB/SUB performance of Tarantool is comparable with Redis (10-20% worse according to our internal benchmarks to be exact, but that's pretty much the same).
\nFor example, let's look at Centrifugo benchmark where we recover zero messages (i.e. emulate a situations when many connections disconnected for a very short time interval due to load balancer reload).
\nFor Redis engine:
\nBenchmarkRedisRecover 26883 ns/op\t 1204 B/op\t 28 allocs/op
Compare it with the same operation measured with Tarantool engine:
\nBenchmarkTarantoolRecover 6292 ns/op\t 563 B/op\t 10 allocs/op
Tarantool can provide new storage properties (like synchronous replication), new adoption. We are pretty excited about adding it as an option.
\nThe reason why Tarantool support is experimental is because Tarantool integration involves one more moving piece – the Centrifuge Lua module which should be run by a Tarantool server.
\nThis increases deployment complexity and given the fact that many users have their own best practices in Tarantool deployment we are still evaluating a sufficient way to distribute Lua part. For now, we are targeting standalone (see examples in centrifugal/tarantool-centrifuge) and Cartridge Tarantool setups (with centrifugal/rotor).
\nRefer to the Tarantool Engine documentation for more details.
\nCentrifugo can now transform events received over persistent connections from users into GRPC calls to the application backend (in addition to the HTTP proxy available in v2).
\nGRPC support should make Centrifugo ready for today's microservice architecture where GRPC is a huge player for inter-service communication.
\nSo we mostly just provide more choices for Centrifugo users here. GRPC has some good advantages – for example an application backend RPC layer which is responsible for communication with Centrifugo can now be generated from Protobuf definitions for all popular programming languages.
\n![](\"https://centrifugal.dev/img/test-tube.svg\")
\nCentrifugo v3 has some valuable server API improvements.
\nThe new subscribe API method allows subscribing connection to a channel at any point in time. This works by utilizing server-side subscriptions. So it's not only possible to subscribe connection to a list of server-side channels during the connection establishment phase – but also later during the connection lifetime. This may be very useful for the unidirectional approach - by emulating client-side subscribe call over request to application backend which in turn calls subscribe Centrifugo server API.
Publish API now returns the current top stream position (offset and epoch) for channels with history enabled.
\nServer history API inherited iteration possibilities described above.
\nChannels command now returns a number of clients in a channel, also supports channel filtering by a pattern. Since we changed how channels call implemented internally there is no limitation anymore to call it when using Redis cluster.
\nAdmin web UI has been updated too to support new API methods, so you can play with new API from its actions tab.
Centrifugo behaves a bit better in cluster mode: as soon as a node leaves a cluster gracefully (upon graceful termination) it sends a shutdown signal to the control channel thus giving other nodes a chance to immediately delete that node from the local registry.
\nWhile preparing the v3 release we improved client connectors too. All existing client connectors now actualized to the latest protocol, support server-side subscriptions, history API.
\nOne important detail is that it's not required to set ?format=protobuf URL param now when connecting to Centrifugo from mobile devices - this is now managed internally by using the WebSocket subprotocol mechanism (requires using the latest client connector version and Centrifugo v3).
You are reading this post on a new project site. It's built with amazing Docusaurus.
\nA lot of documents were actualized, extended, and rewritten. We also now have new chapters like:
\n\nServer API and proxy documentation have been improved significantly.
\n![](\"https://centrifugal.dev/img/stopwatch.svg\")
\nCentrifugo v3 has some notable performance improvements.
\nJSON client protocol now utilizes a couple of libraries (easyjson for encoding and segmentio/encoding for unmarshaling). Actually we use a slightly customized version of easyjson library to achieve even faster performance than it provides out-of-the-box. Changes allowed to speed up JSON encoding and decoding up to 4-5x for small messages. For large payloads speed up can be even more noticeable – we observed up to 30x performance boost when serializing 5kb messages.
For example, let's look at a JSON serialization benchmark result for 256 byte payload. Here is what we had before:
\ncpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkMarshal-12 \t 5883 ns/op\t 1121 B/op\t 6 allocs/op
BenchmarkMarshalParallel-12 \t 1009 ns/op\t 1121 B/op\t 6 allocs/op
BenchmarkUnmarshal-12 \t 1717 ns/op\t 1328 B/op\t 16 allocs/op
BenchmarkUnmarshalParallel-12 \t492.2 ns/op\t 1328 B/op\t 16 allocs/op
And what we have now with mentioned JSON optimizations:
\ncpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkMarshal-12 \t 461.3 ns/op\t 928 B/op\t 3 allocs/op
BenchmarkMarshalParallel-12 \t 250.6 ns/op\t 928 B/op\t 3 allocs/op
BenchmarkUnmarshal-12 \t 476.5 ns/op\t 136 B/op\t 3 allocs/op
BenchmarkUnmarshalParallel-12 \t 107.2 ns/op\t 136 B/op\t 3 allocs/op
Centrifugo Protobuf protocol is still faster than JSON for encoding/decoding on a server-side.
Of course, JSON encoding is only one part of Centrifugo – so you should not expect overall 4x performance improvement. But loaded setups should notice the difference and this should also be a good thing for reducing garbage collection pauses.
\nCentrifugo inherited a couple of other improvements from the Centrifuge library.
\nIn-memory connection hub is now sharded – this should reduce lock contention between operations in different channels. In our artificial benchmarks we noticed a 3x better hub throughput, but in reality the benefit is heavily depends on the usage pattern.
\nCentrifugo now allocates less during message broadcasting to a large number of subscribers.
\nAlso, an upgrade to Go 1.17 for builds results in ~5% performance boost overall, thanks to a new way of passing function arguments and results using registers instead of the stack introduced in Go 1.17.
\nThe final notable thing is an introduction of Centrifugo PRO. This is an extended version of Centrifugo built on top of the OSS version. It provides some unique features targeting business adopters.
\nThose who followed Centrifugo for a long time know that there were some attempts to make project development sustainable. Buy me a coffee and Opencollective approaches were not successful, during a year we got ~300$ of total contributions. While we appreciate these contributions a lot - this does not fairly justify a time spent on Centrifugo maintenance these days and does not allow bringing it to the next level. So here is an another attempt to monetize Centrifugo.
\nCentrifugo PRO details and features described here in docs. Let's see how it goes. We believe that a set of additional functionality can provide great advantages for both small and large-scale Centrifugo setups. PRO features can give useful insights on a system, protect from client API misusing, reduce server resource usage, and more.
\nPRO version will be released soon after Centrifugo v3 OSS.
\nThere are some other changes introduced in v3 but not mentioned here. The full list can be found in the release notes and the migration guide.
\nHope we stepped into an exciting time of the v3 life cycle and many improvements will follow. Join our communities in Telegram and Discord if you have questions or want to follow Centrifugo development:
\n
Enjoy Centrifugo v3, and let the Centrifugal force be with you.
\nSpecial thanks to Anton Silischev for the help with v3 tests, examples and CI. To Leon Sorokin for the spinning CSS Centrifugo logo. To Michael Filonenko for the help with Tarantool. To German Saprykin for Dart magic.
Thanks to the community members who tested out Centrifugo v3 beta, found bugs and sent improvements.
In this post I'll try to introduce Centrifuge - the heart of Centrifugo.
\nCentrifuge is a real-time messaging library for the Go language.
\nThis post is going to be pretty long (looks like I am a huge fan of long reads) – so make sure you also have a drink (probably two) and let's go!
\nI wrote several blog posts before (for example this one – yep, it's on Medium...) about an original motivation of Centrifugo server.
\nCentrifugo server is not the same as Centrifuge library for Go. It's a full-featured project built on top of Centrifuge library. Naming can be confusing, but it's not too hard once you spend some time with ecosystem.
In short – Centrifugo was implemented to help traditional web frameworks dealing with many persistent connections (like WebSocket or SockJS HTTP transports). So frameworks like Django or Ruby on Rails, or frameworks from the PHP world could be used on a backend but still provide real-time messaging features like chats, multiplayer browser games, etc for users. With a little help from Centrifugo.
\nNow there are cases when Centrifugo server used in conjunction even with a backend written in Go. While Go mostly has no problems dealing with many concurrent connections – Centrifugo provides some features beyond simple message passing between a client and a server. That makes it useful, especially since design is pretty non-obtrusive and fits well microservices world. Centrifugo is used in some well-known projects (like ManyChat, Yoola.io, Spot.im, Badoo etc).
\nAt the end of 2018, I released Centrifugo v2 based on a real-time messaging library for Go language – Centrifuge – the subject of this post.
\nIt was a pretty hard experience to decouple Centrifuge out of the monolithic Centrifugo server – I was unable to make all the things right immediately, so Centrifuge library API went through several iterations where I introduced backward-incompatible changes. All those changes targeted to make Centrifuge a more generic tool and remove opinionated or limiting parts.
\nThis is ... well, a framework to build real-time messaging applications with Go language. If you ever heard about socket.io – then you can think about Centrifuge as an analogue. I think the most popular applications these days are chats of different forms, but I want to emphasize that Centrifuge is not a framework to build chats – it's a generic instrument that can be used to create different sorts of real-time applications – real-time charts, multiplayer games.
\nThe obvious choice for real-time messaging transport to achieve fast and cross-platform bidirectional communication these days is WebSocket. Especially if you are targeting a browser environment. You mostly don't need to use WebSocket HTTP polyfills in 2021 (though there are still corner cases so Centrifuge supports SockJS polyfill).
\nCentrifuge has its own custom protocol on top of plain WebSocket or SockJS frames.
\nThe reason why Centrifuge has its own protocol on top of underlying transport is that it provides several useful primitives to build real-time applications. The protocol described as strict Protobuf schema. It's possible to pass JSON or binary Protobuf-encoded data over the wire with Centrifuge.
\nGRPC is very handy these days too (and can be used in a browser with a help of additional proxies), some developers prefer using it for real-time messaging apps – especially when one-way communication needed. It can be a bit better from integration perspective but more resource-consuming on server side and a bit trickier to deploy.
Take a look at WebTransport – a brand-new spec for web browsers to allow fast communication between a client and a server on top of QUIC – it may be a good alternative to WebSocket in the future. This in a draft status at the moment, but it's already possible to play with in Chrome.
Own protocol is one of the things that prove the framework status of Centrifuge. This dictates certain limits (for example, you can't just use an alternative message encoding) and makes developers use custom client connectors on a front-end side to communicate with a Centrifuge-based server (see more about connectors in ecosystem part).
\nBut protocol solves many practical tasks – and here we are going to look at real-time features it provides for a developer.
\nTo start working with Centrifuge you need to start Centrifuge server Node. Node is a core of Centrifuge – it has many useful methods – set event handlers, publish messages to channels, etc. We will look at some events and channels concept very soon.
\nAlso, Node abstracts away scalability aspects, so you don't need to think about how to scale WebSocket connections over different server instances and still have a way to deliver published messages to interested clients.
\nFor now, let's start a single instance of Node that will serve connections for us:
\nnode, err := centrifuge.New(centrifuge.DefaultConfig)
if err != nil {
log.Fatal(err)
}
if err := node.Run(); err != nil {
log.Fatal(err)
}
It's also required to serve a WebSocket handler – this is possible just by registering centrifuge.WebsocketHandler in HTTP mux:
wsHandler := centrifuge.NewWebsocketHandler(node, centrifuge.WebsocketConfig{})
http.Handle(\"/connection/websocket\", wsHandler)
Now it's possible to connect to a server (using Centrifuge connector for a browser called centrifuge-js):
const centrifuge = new Centrifuge('ws://localhost:8000/connection/websocket');
centrifuge.connect();
Though connection will be rejected by the server since we also need to provide authentication details – Centrifuge expects explicitly provided connection Credentials to accept connection.
Let's look at how we can tell Centrifuge details about connected user identity, so it could accept an incoming connection.
\nThere are two main ways to authenticate client connection in Centrifuge.
\nThe first one is over the native middleware mechanism. It's possible to wrap centrifuge.WebsocketHandler or centrifuge.SockjsHandler with middleware that checks user authentication and tells Centrifuge current user ID over context.Context:
func auth(h http.Handler) http.Handler {
\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
\t\tcred := ¢rifuge.Credentials{
\t\t\tUserID: \"42\",
\t\t}
\t\tnewCtx := centrifuge.SetCredentials(r.Context(), cred)
\t\tr = r.WithContext(newCtx)
\t\th.ServeHTTP(w, r)
\t})
}
So WebsocketHandler can be registered this way (note that a handler now wrapped by auth middleware):
\nwsHandler := centrifuge.NewWebsocketHandler(node, centrifuge.WebsocketConfig{})
http.Handle(\"/connection/websocket\", auth(wsHandler))
Another authentication way is a bit more generic – developers can authenticate connection based on custom token sent from a client inside first WebSocket/SockJS frame. This is called connect frame in terms of Centrifuge protocol. Any string token can be set – this opens a way to use JWT, Paceto, and any other kind of authentication tokens. For example see an authenticaton with JWT.
BTW it's also possible to pass any information from client side with a first connect message from client to server and return custom information about server state to a client. This is out of post scope though.
Nothing prevents you to integrate Centrifuge with OAuth2 or another framework session mechanism – like Gin for example.
\nAs soon as a client connected and successfully authenticated it can subscribe to channels. Channel (room or topic in other systems) is a lightweight and ephemeral entity in Centrifuge. Channel can have different features (we will look at some channel features below). Channels created automatically as soon as the first subscriber joins and destroyed as soon as the last subscriber left.
\nThe application can have many real-time features – even on one app screen. So sometimes client subscribes to several channels – each related to a specific real-time feature (for example one channel for chat updates, one channel likes notification stream, etc).
\nChannel is just an ASCII string. A developer is responsible to find the best channel naming convention suitable for an application. Channel naming convention is an important aspect since in many cases developers want to authorize subscription to a channel on the server side – so only authorized users could listen to specific channel updates.
\nLet's look at a basic subscription example on the client-side:
\ncentrifuge.subscribe('example', function(msgCtx) {
console.log(msgCtx)
})
On the server-side, you need to define subscribe event handler. If subscribe event handler not set then the connection won't be able to subscribe to channels at all. Subscribe event handler is where a developer may check permissions of the current connection to read channel updates. Here is a basic example of subscribe event handler that simply allows subscriptions to channel example for all authenticated connections and reject subscriptions to all other channels:
node.OnConnect(func(client *centrifuge.Client) {
client.OnSubscribe(func(e centrifuge.SubscribeEvent, cb centrifuge.SubscribeCallback) {
if e.Channel != \"example\" {
cb(centrifuge.SubscribeReply{}, centrifuge.ErrorPermissionDenied)
return
}
cb(centrifuge.SubscribeReply{}, nil)
})
})
You may notice a callback style of reacting to connection related things. While not being very idiomatic for Go it's very practical actually. The reason why we use callback style inside client event handlers is that it gives a developer possibility to control operation concurrency (i.e. process sth in separate goroutines or goroutine pool) and still control the order of events. See an example that demonstrates concurrency control in action.
\nNow if some event published to a channel:
\n// Here is how we can publish data to a channel.
node.Publish(\"example\", []byte(`{\"input\": \"hello\"}`))
– data will be delivered to a subscribed client, and message will be printed to Javascript console. PUB/SUB in its usual form.
\nThough Centrifuge protocol based on Protobuf schema in example above we published a JSON message into a channel. By default, we can only send JSON to connections since default protocol format is JSON. But we can switch to Protobuf-based binary protocol by connecting to ws://localhost:8000/connection/websocket?format=protobuf endpoint – then it's possible to send binary data to clients.
While Centrifuge mostly shines when you need channel semantics it's also possible to send any data to connection directly – to achieve bidirectional asynchronous communication, just what a native WebSocket provides.
\nTo send a message to a server one can use the send method on the client-side:
centrifuge.send({\"input\": \"hello\"});
On the server-side data will be available inside a message handler:
\nclient.OnMessage(func(e centrifuge.MessageEvent) {
log.Printf(\"message from client: %s\", e.Data)
})
And vice-versa, to send data to a client use Send method of centrifuge.Client:
client.Send([]byte(`{\"input\": \"hello\"}`))
To listen to it on the client-side:
\ncentrifuge.on('message', function(data) {
console.log(data);
});
RPC is a primitive for sending a request from a client to a server and waiting for a response (in this case all communication still happens via asynchronous message passing internally, but Centrifuge takes care of matching response data to request previously sent).
\nOn client side it's as simple as:
\nconst resp = await centrifuge.rpc('my_method', {});
On server side RPC event handler should be set to make calls available:
\nclient.OnRPC(func(e centrifuge.RPCEvent, cb centrifuge.RPCCallback) {
if e.Method == \"my_method\" {
cb(centrifuge.RPCReply{Data: []byte(`{\"result\": \"42\"}`)}, nil)
return
}
cb(centrifuge.RPCReply{}, centrifuge.ErrorMethodNotFound)
})
Note, that it's possible to pass the name of RPC and depending on it and custom request params return different results to a client – just like a regular HTTP request but over asynchronous WebSocket (or SockJS) connection.
\nIn many cases, a client is a source of knowledge which channels it wants to subscribe to on a specific application screen. But sometimes you want to control subscriptions to channels on a server-side. This is also possible in Centrifuge.
\nIt's possible to provide a slice of channels to subscribe connection to at the moment of connection establishment phase:
\nnode.OnConnecting(func(ctx context.Context, e centrifuge.ConnectEvent) (centrifuge.ConnectReply, error) {
return centrifuge.ConnectReply{
Subscriptions: map[string]centrifuge.SubscribeOptions{
\"example\": {},
},
}, nil
})
Note, that OnConnecting does not follow callback-style – this is because it can only happen once at the start of each connection – so there is no need to control operation concurrency.
In this case on the client-side you will have access to messages published to channels by listening to on('publish') event:
centrifuge.on('publish', function(msgCtx) {
console.log(msgCtx);
});
Also, centrifuge.Client has Subscribe and Unsubscribe methods so it's possible to subscribe/unsubscribe client to/from channel somewhere in the middle of its long WebSocket session.
Every time a message published to a channel it's possible to provide custom history options. For example:
\nnode.Publish(
\"example\",
[]byte(`{\"input\": \"hello\"}`),
centrifuge.WithHistory(300, time.Minute),
)
In this case, Centrifuge will maintain a windowed Publication cache for a channel - or in other words, maintain a publication stream. This stream will have time retention (one minute in the example above) and the maximum size will be limited to the value provided during Publish (300 in the example above).
\nEvery message inside a history stream has an incremental offset field. Also, a stream has a field called epoch – this is a unique identifier of stream generation - thus client will have a possibility to distinguish situations where a stream is completely removed and there is no guarantee that no messages have been lost in between even if offset looks fine.
Client protocol provides a possibility to paginate over a stream from a certain position with a limit:
\nconst streamPosition = {'offset': 0, epoch: 'xyz'}
resp = await sub.history({since: streamPosition, limit: 10});
Iteration over history stream is a new feature which is just merged into Centrifuge master branch and can only be used from Javascript client at the moment.
\nAlso, Centrifuge has an automatic message recovery feature. Automatic recovery is very useful in scenarios when tons of persistent connections start reconnecting at once. I already described why this is useful in one of my previous posts about Websocket scalability. In short – since WebSocket connections are stateful then at the moment of mass reconnect they can create a very big spike in load on your main application database. Such mass reconnects are a usual thing in practice - for example when you reload your load balancers or re-deploying the Websocket server (new code version).
\nOf course, recovery can also be useful for regular short network disconnects - when a user travels in the subway for example. But you always need a way to load an actual state from the main application database in case of an unsuccessful recovery.
\nTo enable automatic recovery you can provide the Recover flag in subscribe options:
client.OnSubscribe(func(e centrifuge.SubscribeEvent, cb centrifuge.SubscribeCallback) {
cb(centrifuge.SubscribeReply{
Options: centrifuge.SubscribeOptions{
Recover: true,
},
}, nil)
})
Obviously, recovery will work only for channels where history stream maintained. The limitation in recovery is that all missed publications sent to client in one protocol frame – pagination is not supported during recovery process. This means that recovery is mostly effective for not too long offline time without tons of missed messages.
\nAnother cool thing Centrifuge exposes to developers is online presence information for channels. Presence information contains a list of active channel subscribers. This is useful to show the online status of players in a game for example.
\nAlso, it's possible to turn on Join/Leave message feature inside channels: so each time connection subscribes to a channel all channel subscribers receive a Join message with client information (client ID, user ID). As soon as the client unsubscribes Leave message is sent to remaining channel subscribers with information who left a channel.
\nHere is how to enable both online presence and join/leave features for a subscription to channel:
\nclient.OnSubscribe(func(e centrifuge.SubscribeEvent, cb centrifuge.SubscribeCallback) {
cb(centrifuge.SubscribeReply{
Options: centrifuge.SubscribeOptions{
Presence: true,
JoinLeave: true,
},
}, nil)
})
On a client-side then it's possible to call for the presence and setting event handler for join/leave messages.
\nThe important thing to be aware of when using Join/Leave messages is that this feature can dramatically increase CPU utilization and overall traffic in channels with a big number of active subscribers – since on every client connect/disconnect event such Join or Leave message must be sent to all subscribers. The advice here – avoid using Join/Leave messages or be ready to scale (Join/Leave messages scale well when adding more Centrifuge Nodes – more about scalability below).
\nOne more thing to remember is that online presence information can also be pretty expensive to request in channels with many active subscribers – since it returns information about all connections – thus payload in response can be large. To help a bit with this situation Centrifuge has a presence stats client API method. Presence stats only contain two counters: the number of active connections in the channel and amount of unique users in the channel.
\nIf you still need to somehow process online presence in rooms with a massive number of active subscribers – then I think you better do it in near real-time - for example with fast OLAP like ClickHouse.
\nTo be fair it's not too hard to implement most of the features above inside one in-memory process. Yes, it takes time, but the code is mostly straightforward. When it comes to scalability things tend to be a bit harder.
\nCentrifuge designed with the idea in mind that one machine is not enough to handle all application WebSocket connections. Connections should scale over application backend instances, and it should be simple to add more application nodes when the amount of users (connections) grows.
\nCentrifuge abstracts scalability over the Node instance and two interfaces: Broker interface and PresenceManager interface.
A broker is responsible for PUB/SUB and streaming semantics:
\ntype Broker interface {
\tRun(BrokerEventHandler) error
\tSubscribe(ch string) error
\tUnsubscribe(ch string) error
\tPublish(ch string, data []byte, opts PublishOptions) (StreamPosition, error)
\tPublishJoin(ch string, info *ClientInfo) error
\tPublishLeave(ch string, info *ClientInfo) error
\tPublishControl(data []byte, nodeID string) error
\tHistory(ch string, filter HistoryFilter) ([]*Publication, StreamPosition, error)
\tRemoveHistory(ch string) error
}
See full version with comments in source code.
\nEvery Centrifuge Node subscribes to channels via a broker. This provides a possibility to scale connections over many node instances – published messages will flow only to nodes with active channel subscribers.
\nIt's and important thing to combine PUB/SUB with history inside a Broker implementation to achieve an atomicity of saving message into history stream and publishing it to PUB/SUB with generated offset.
\nPresenceManager is responsible for online presence information management:
\ntype PresenceManager interface {
\tPresence(ch string) (map[string]*ClientInfo, error)
\tPresenceStats(ch string) (PresenceStats, error)
\tAddPresence(ch string, clientID string, info *ClientInfo, expire time.Duration) error
\tRemovePresence(ch string, clientID string) error
}
Broker and PresenceManager together form an Engine interface:
type Engine interface {
\tBroker
\tPresenceManager
}
By default, Centrifuge uses MemoryEngine that does not use any external services but limits developers to using only one Centrifuge Node (i.e. one server instance). Memory Engine is fast and can be suitable for some scenarios - even in production (with configured backup instance) – but as soon as the number of connections grows – you may need to load balance connections to different server instances. Here comes the Redis Engine.
Redis Engine utilizes Redis for Broker and PresenceManager parts.
\nHistory cache saved to Redis STREAM or Redis LIST data structures. For presence, Centrifuge uses a combination of HASH and ZSET structures.
\nCentrifuge tries to fully utilize the connection between Node and Redis by using pipelining where possible and smart batching technique. All operations done in a single RTT with the help of Lua scripts loaded automatically to Redis on engine start.
\nRedis is pretty fast and will allow your app to scale to some limits. When Redis starts being a bottleneck it's possible to shard data over different Redis instances. Client-side consistent sharding is built-in in Centrifuge and allows scaling further.
\nIt's also possible to achieve Redis's high availability with built-in Sentinel support. Redis Cluster supported too. So Redis Engine covers many options to communicate with Redis deployed in different ways.
\nAt Avito we served about 800k active connections in the messenger app with ease using a slightly adapted Centrifuge Redis Engine, so an approach proved to be working for rather big applications. We will look at some more concrete numbers below in the performance section.
\nBoth Broker and PresenceManager are pluggable, so it's possible to replace them with alternative implementations. Examples show how to use Nats server for at most once only PUB/SUB together with Centrifuge. Also, we have an example of full-featured Engine for Tarantool database – Tarantool Engine shows even better throughput for history and presence operations than Redis-based Engine (up to 10x for some ops).
Since Centrifuge is a messaging system I also want to describe its order and message delivery guarantees.
\nMessage ordering in channels supported. As soon as you publish messages into channels one after another of course.
\nMessage delivery model is at most once by default. This is mostly comes from PUB/SUB model – message can be dropped on Centrifuge level if subscriber is offline or simply on broker level – since Redis PUB/SUB also works with at most once guarantee.
\nThough if you maintain history stream inside a channel then things become a bit different. In this case you can tell Centrifuge to check client position inside stream. Since every publication has a unique incremental offset Centrifuge can track that client has correct offset inside a channel stream. If Centrifuge detects any missed messages it disconnects a client with special code – thus make it reconnect and recover messages from history stream. Since a message first saved to history stream and then published to PUB/SUB inside broker these mechanisms allow achieving at least once message delivery guarantee.
\n![\"What](\"https://i.imgur.com/PLb9xS5.jpg\")
Even if stream completely expired or dropped from broker memory Centrifuge will give a client a tip that messages could be lost – so client has a chance to restore state from a main application database.
\nHere I want to be fair with my readers – Centrifuge is not ideal. This is a project maintained mostly by one person at the moment with all consequences. This hits an ecosystem a lot, can make some design choices opinionated or non-optimal.
\nI mentioned in the first post that Centrifuge built on top of the custom protocol. The protocol is based on a strict Protobuf schema, works with JSON and binary data transfer, supports many features. But – this means that to connect to the Centrifuge-based server developers have to use custom connectors that can speak with Centrifuge over its custom protocol.
\nThe difficulty here is that protocol is asynchronous. Asynchronous protocols are harder to implement than synchronous ones. Multiplexing frames allows achieving good performance and fully utilize a single connection – but it hurts simplicity.
\nAt this moment Centrifuge has client connectors for:
\nNot all clients support all protocol features. Another drawback is that all clients do not have a persistent maintainer – I mostly maintain everything myself. Connectors can have non-idiomatic and pretty dumb code since I had no previous experience with mobile development, they lack proper tests and documentation. This is unfortunate.
\nThe good thing is that all connectors feel very similar, I am quickly releasing new versions when someone sends a pull request with improvements or bug fixes. So all connectors are alive.
\nI maintain a feature matrix in connectors to let users understand what's supported. Actually feature support is pretty nice throughout all these connectors - there are only several things missing and not so much work required to make all connectors full-featured. But I really need help here.
\nIt will be a big mistake to not mention Centrifugo as a big plus for Centrifuge library ecosystem. Centrifugo is a server deployed in many projects throughout the world. Many features of Centrifuge library and its connectors have already been tested by Centrifugo users.
\nOne more thing to mention is that Centrifuge does not have v1 release. It still evolves – I believe that the most dramatic changes have already been made and backward compatibility issues will be minimal in the next releases – but can't say for sure.
\nI made a test stand in Kubernetes with one million connections.
\nI can't call this a proper benchmark – since in a benchmark your main goal is to destroy a system, in my test I just achieved some reasonable numbers on limited hardware. These numbers should give a good insight into a possible throughput, latency, and estimate hardware requirements (at least approximately).
\nConnections landed on different server pods, 5 Redis instances have been used to scale connections between pods.
\nThe detailed test stand description can be found in Centrifugo documentation.
\n![\"Benchmark\"](\"https://centrifugal.dev/assets/images/benchmark-b670972866abdc8936d2aef84333dd0c.gif\")
Some quick conclusions are:
\nCentrifuge does not allow subscribing on the same channel twice inside a single connection. It's not simple to add due to design decisions made – though there was no single user report about this in seven years of Centrifugo/Centrifuge history.
\nCentrifuge does not support wildcard subscriptions. Not only because I never needed this myself but also due to some design choices made – so be aware of this.
\nSockJS fallback does not support binary data - only JSON. If you want to use binary in your application then you can only use WebSocket with Centrifuge - there is no built-in fallback transport in this case.
\nSockJS also requires sticky session support from your load balancer to emulate a stateful bidirectional connection with its HTTP fallback transports. Ideally, Centrifuge will go away from SockJS at some point, maybe when WebTransport becomes mature so users will have a choice between WebTransport or WebSocket.
\nWebsocket permessage-deflate compression supported (thanks to Gorilla WebSocket), but it can be pretty expensive in terms of CPU utilization and memory usage – the overhead depends on usage pattern, it's pretty hard to estimate in numbers.
As said above you cannot only rely on Centrifuge for state recovery – it's still required to have a way to fully load application state from the main database.
\nAlso, I am not very happy with current error and disconnect handling throughout the connector ecosystem – this can be improved though, and I have some ideas for the future.
\nI am adding examples to _examples folder of Centrifuge repo. These examples completely cover Centrifuge API - including things not mentioned here.
\nCheck out the tips & tricks section of README – it contains some additional insights about an implementation.
\nI think Centrifuge could be a nice alternative to socket.io - with a better performance, main server implementation in Go language, and even more builtin features to build real-time apps.
\nCentrifuge ecosystem definitely needs more work, especially in client connectors area, tutorials, community, stabilizing API, etc.
\nCentrifuge fits pretty well proprietary application development where time matters and deadlines are close, so developers tend to choose a ready solution instead of writing their own. I believe Centrifuge can be a great time saver here.
\nFor Centrifugo server users Centrifuge package provides a way to write a more flexible server code adapted for business requirements but still use the same real-time core and have the same protocol features.
", "url": "https://centrifugal.dev/blog/2021/01/15/centrifuge-intro", "title": "Centrifuge – real-time messaging with Go", "summary": "An introduction to Centrifuge – real-time messaging with Go", "date_modified": "2021-01-15T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "centrifuge", "go" ] }, { "id": "https://centrifugal.dev/blog/2020/11/12/scaling-websocket", "content_html": "I believe that in 2020 WebSocket is still an entertaining technology which is not so well-known and understood like HTTP. In this blog post I'd like to tell about state of WebSocket in Go language ecosystem, and a way we could write scalable WebSocket servers with Go and beyond Go.
\nWe won't talk a lot about WebSocket transport pros and cons – I'll provide links to other resources on this topic. Most advices here are generic enough and can be easily approximated to other programming languages. Also in this post we won't talk about ready to use solutions (if you are looking for it – check out Real-time Web Technologies guide by Phil Leggetter), just general considerations. There is not so much information about scaling WebSocket on the internet so if you are interested in WebSocket and real-time messaging technologies - keep on reading.
\nIf you don't know what WebSocket is – check out the following curious links:
\nAs soon as you know WebSocket basics – we can proceed.
\nSpeaking about scalable servers that work with many persistent WebSocket connections – I found several important tasks such a server should be able to do:
\nOf course not all of these points equally important in various situations.
Below we will look at some tips which relate to these points.
\n![\"one_hour_scale\"](\"https://i.imgur.com/4lYjJSP.png\")
In Go language ecosystem we have several libraries which can be used as a building block for a WebSocket server.
\nPackage golang.org/x/net/websocket is considered deprecated.
\nThe default choice in the community is gorilla/websocket library. Made by Gary Burd (who also gifted us an awesome Redigo package to communicate with Redis) – it's widely used, performs well, has a very good API – so in most cases you should go with it. Some people think that library not actively maintained at moment – but this is not quite true, it implements full WebSocket RFC, so actually it can be considered done.
\nIn 2018 my ex-colleague Sergey Kamardin open-sourced gobwas/ws library. It provides a bit lower-level API than gorilla/websocket thus allows reducing RAM usage per connection and has nice optimizations for WebSocket upgrade process. It does not support WebSocket permessage-deflate compression but otherwise a good alternative you can consider using. If you have not read Sergey's famous post A Million WebSockets and Go – make a bookmark!
One more library is nhooyr/websocket. It's the youngest one and actively maintained. It compiles to WASM which can be a cool thing for someone. The API is a bit different from what gorilla/websocket offers, and one of the big advantages I see is that it solves a problem with a proper WebSocket closing handshake which is a bit hard to do right with Gorilla WebSocket.
You can consider all listed libraries except one from x/net for your project. Take a library, follow its examples (make attention to goroutine-safety of various API operations). Personally I prefer Gorilla WebSocket at moment since it's feature-complete and battle tested by tons of projects around Go world.
OK, so you have chosen a library and built a server on top of it. As soon as you put it in production the interesting things start happening.
\nLet's start with several OS specific key things you should do to prepare for many connections from WebSocket clients.
\nEvery connection will cost you an open file descriptor, so you should tune a maximum number of open file descriptors your process can use. An errors like too many open files raise due to OS limit on file descriptors which is usually 256-1024 by default (see with ulimit -n on Unix). A nice overview on how to do this on different systems can be found in Riak docs. Wanna more connections? Make this limit higher.
Nice tip here is to limit a maximum number of connections your process can serve – making it less than known file descriptor limit:
\n// ulimit -n == 65535
if conns.Len() >= 65500 {
return errors.New(\"connection limit reached\")
}
conns.Add(conn)
– otherwise you have a risk to not even able to look at pprof when things go bad. And you always need monitoring of open file descriptors.
You can also consider using netutil.LimitListener for this task, but don't forget to put pprof on another port with another HTTP server instance in this case.
\nKeep attention on Ephemeral ports problem which is often happens between your load balancer and your WebSocket server. The problem arises due to the fact that each TCP connection uniquely identified in the OS by the 4-part-tuple:
\nsource ip | source port | destination ip | destination port
On balancer/server boundary you are limited in 65536 possible variants by default. But actually due to some OS limits and sockets in TIME_WAIT state the number is even less. A very good explanation and how to deal with it can be found in Pusher blog.
\nYour possible number of connections also limited by conntrack table. Netfilter framework which is part of iptables keeps information about all connections and has limited size for this information. See how to see its limits and instructions to increase in this article.
\nOne more thing you can do is tune your network stack for performance. Do this only if you understand that you need it. Maybe start with this gist, but don't optimize without full understanding why you are doing this.
\nNow let's speak about sending many messages. The general tips follows.
\nMake payload smaller. This is obvious – fewer data means more effective work on all layers. BTW WebSocket framing overhead is minimal and adds only 2-8 bytes to your payload. You can read detailed dedicated research in Dissecting WebSocket's Overhead article. You can reduce an amount of data traveling over network with permessage-deflate WebSocket extension, so your data will be compressed. Though using permessage-deflate is not always a good thing for server due to poor performance of flate, so you should be prepared for a CPU and RAM resource usage on server side. While Gorilla WebSocket has a lot of optimizations internally by reusing flate writers, overhead is still noticeable. The increase value heavily depends on your load profile.
Make less system calls. Every syscall will have a constant overhead, and actually in WebSocket server under load you will mostly see read and write system calls in your CPU profiles. An advice here – try to use client-server protocol that supports message batching, so you can join individual messages together.
\nUse effective message serialization protocol. Maybe use code generation for JSON to avoid extensive usage of reflect package done by Go std lib. Maybe use sth like gogo/protobuf package which allows to speedup Protobuf marshalling and unmarshalling. Unfortunately Gogo Protobuf is going through hard times\n at this moment. Try to serialize a message only once when sending to many subscribers.
\nHave a way to scale to several machines - more power, more possible messages. We will talk about this very soon.
\n![\"ie\"](\"https://i.imgur.com/IAOyvmg.png\")
Even in 2020 there are still users which cannot establish connection with WebSocket server. Actually the problem mostly appears with browsers. Some users still use old browsers. But they have a choice – install a newer browser. Still, there could also be users behind corporate proxies. Employees can have a trusted certificate installed on their machine so company proxy can re-encrypt even TLS traffic. Also, some browser extensions can block WebSocket traffic.
\nOne ready solution to this is Sockjs-Go library. This is a mature library that provides fallback transport for WebSocket. If client does not succeed with WebSocket connection establishment then client can use some of HTTP transports for client-server communication: EventSource aka Server-Sent Events, XHR-streaming, Long-Polling etc. The downside with those transports is that to achieve bidirectional communication you should use sticky sessions on your load balancer since SockJS keeps connection session state in process memory. We will talk about many instances of your WebSocket server very soon.
\nYou can implement WebSocket fallback yourself, this should be simple if you have a sliding window message stream on your backend which we will discuss very soon.
\nMaybe look at GRPC, depending on application it could be better or worse than WebSocket – in general you can expect a better performance and less resource consumption from WebSocket for bidirectional communication case. My measurements for a bidirectional scenario showed 3x win for WebSocket (binary + GOGO protobuf) in terms of server CPU consumption and 4 times less RAM per connection. Though if you only need RPC then GRPC can be a better choice. But you need additional proxy to work with GRPC from a browser.
\nYou can optimize client-server protocol, tune your OS, but at some point you won't be able to use only one process on one server machine. You need to scale connections and work your server does over different server machines. Horizontal scaling is also good for a server high availability. Actually there are some sort of real-time applications where a single isolated process makes sense - for example multiplayer games where limited number of players play independent game rounds.
\n![\"many_instances\"](\"https://i.imgur.com/8ElqpjI.png\")
As soon as you distribute connections over several machines you have to find a way to deliver a message to a certain user. The basic approach here is to publish messages to all server instances. This can work but this does not scale well. You need a sort of instance discovery to make this less painful.
\nHere comes PUB/SUB, where you can connect WebSocket server instances over central PUB/SUB broker. Clients that establish connections with your WebSocket server subscribe to topics (channels) in a broker, and as soon as you publish a message to that topic it will be delivered to all active subscribers on WebSocket server instances. If server node does not have interested subscriber then it won't get a message from a broker thus you are getting effective network communication.
\nActually the main picture of this post illustrates exactly this architecture:
\n
Let's think about requirements for a broker for real-time messaging application. We want a broker:
\nPersonally when we talk about such brokers here are some options that come into my mind:
\n\nSure there are more exist including libraries like ZeroMQ or nanomsg.
\nBelow I'll try to consider these solutions for the task of making scalable WebSocket server facing many user connections from Internet.
\nIf you are looking for unreliable at most once PUB/SUB then any of solutions mentioned above should be sufficient. Many real-time messaging apps are ok with at most once guarantee delivery.
\nIf you don't want to miss messages then things are a bit harder. Let's try to evaluate these options for a task where application has lots of different topics from which it wants to receive messages with at least once guarantee (having a personal topic per client is common thing in applications). A short analysis below can be a bit biased, but I believe thoughts are reasonable enough. I did not found enough information on the internet about scaling WebSocket beyond a single server process, so I'll try to fill the gap a little based on my personal knowledge without pretending to be absolutely objective in these considerations.
\nIn some posts on the internet about scaling WebSocket I saw advices to use RabbitMQ for PUB/SUB stuff in real-time messaging server. While this is a great messaging server, it does not like a high rate of queue bind and unbind type of load. It will work, but you will need to use a lot of server resources for not so big number of clients (imagine having millions of queues inside RabbitMQ). I have an example from my practice where RabbitMQ consumed about 70 CPU cores to serve real-time messages for 100k online connections. After replacing it with Redis keeping the same message delivery semantics we got only 0.3 CPU consumption on broker side.
\nKafka and Pulsar are great solutions, but not for this task I believe. The problem is again in dynamic ephemeral nature of our topics. Kafka also likes a more stable configuration of its topics. Keeping messages on disk can be an overkill for real-time messaging task. Also your consumers on Kafka server should pull from millions of different topics, not sure how well it performs, but my thoughts at moment - this should not perform very well. Kafka itself scales perfectly, you will definitely be able to achieve a goal but resource usage will be significant. Here is a post from Trello where they moved from RabbitMQ to Kafka for similar real-time messaging task and got about 5x resource usage improvements. Note also that the more partitions you have the more heavy failover process you get.
\nNats and Nats-Streaming. Raw Nats can only provide at most once guarantee. BTW recently Nats developers released native WebSocket support, so you can consider it for your application. Nats-Streaming server as broker will allow you to not lose messages. To be fair I don't have enough information about how well Nats-Streaming scales to millions of topics. An upcoming Jetstream which will be a part of Nats server can also be an interesting option – like Kafka it provides a persistent stream of messages for at least once delivery semantics. But again, it involves disk storage, a nice thing for backend microservices communication but can be an overkill for real-time messaging task.
\nSure Tarantool can fit to this task well too. It's fast, im-memory and flexible. Some possible problems with Tarantool are not so healthy state of its client libraries, complexity and the fact that it's heavily enterprise-oriented. You should invest enough time to benefit from it, but this can worth it actually. See an article on how to do a performant broker for WebSocket applications with Tarantool.
\nBuilding PUB/SUB system on top of ZeroMQ will require you to build separate broker yourself. This could be an unnecessary complexity for your system. It's possible to implement PUB/SUB pattern with ZeroMQ and nanomsg without a central broker, but in this case messages without active subscribers on a server will be dropped on a consumer side thus all publications will travel to all server nodes.
\nMy personal choice at moment is Redis. While Redis PUB/SUB itself provides at most once guarantee, you can build at least once delivery on top of PUB/SUB and Redis data structures (though this can be challenging enough). Redis is very fast (especially when using pipelining protocol feature), and what is more important – very predictable. It gives you a good understanding of operation time complexity. You can shard topics over different Redis instances running in HA setup - with Sentinel or with Redis Cluster. It allows writing LUA procedures with some advanced logic which can be uploaded over client protocol thus feels like ordinary commands. You can use Redis to keep sliding window event stream which gives you access to missed messages from a certain position. We will talk about this later.
\nOK, the end of opinionated thoughts here :)
\nDepending on your choice the implementation of your system will vary and will have different properties – so try to evaluate possible solutions based on your application requirements. Anyway, whatever broker will be your choice, try to follow this rules to build effective PUB/SUB system:
\n![\"mass_reconnect\"](\"https://i.imgur.com/S9koKYg.png\")
Let's talk about one more problem that is unique for Websocket servers compared to HTTP. Your app can have thousands or millions of active WebSocket connections. In contract to stateless HTTP APIs your application is stateful. It uses push model. As soon as you deploying your WebSocket server or reload your load balancer (Nginx maybe) – connections got dropped and all that army of users start reconnecting. And this can be like an avalanche actually. How to survive?
\nFirst of all - use exponential backoff strategies on client side. I.e. reconnect with intervals like 1, 2, 4, 8, 16 seconds with some random jitter.
\nTurn on various rate limiting strategies on your WebSocket server, some of them should be turned on your backend load balancer level (like controlling TCP connection establishment rate), some are application specific (maybe limit an amount of requests from certain user).
\nOne more interesting technique to survive massive reconnect is using JWT (JSON Web Token) for authentication. I'll try to explain why this can be useful.
\n![\"jwt\"](\"https://i.imgur.com/aaTEhXo.png\")
As soon as your client start reconnecting you will have to authenticate each connection. In massive setups with many persistent connection this can be a very significant load on your Session backend. Since you need an extra request to your session storage for every client coming back. This can be a no problem for some infrastructures but can be really disastrous for others. JWT allows to reduce this spike in load on session storage since it can have all required authentication information inside its payload. When using JWT make sure you have chosen a reasonable JWT expiration time – expiration interval depends on your application nature and just one of trade-offs you should deal with as developer.
\nDon't forget about making an effective connection between your WebSocket server and broker – as soon as all clients start reconnecting you should resubscribe your server nodes to all topics as fast as possible. Use techniques like smart batching at this moment.
\nLet's look at a small piece of code that demonstrates this technique. Imagine we have a source channel from which we get items to process. We don’t want to process items individually but in batch. For this we wait for first item coming from channel, then try to collect as many items from channel buffer as we want without blocking and timeouts involved. And then process slice of items we collected at once. For example build Redis pipeline from them and send to Redis in one connection write call.
\nmaxBatchSize := 50
for {
select {
case item := <-sourceCh:
batch := []string{item}
loop:
for len(batch) < maxBatchSize {
select {
case item := <-sourceCh:
batch = append(batch, item)
default:
break loop
}
}
// Do sth with collected batch of items.
println(len(batch))
}
}
Look at a complete example in a Go playground: https://play.golang.org/p/u7SAGOLmDke.
\nI also made a repo where I demonstrate how this technique together with Redis pipelining feature allows to fully utilize connection for a good performance https://github.com/FZambia/redigo-smart-batching.
\nAnother advice for those who run WebSocket services in Kubernetes. Learn how your ingress behaves – for example Nginx ingress can reload its configuration on every change inside Kubernetes services map resulting into closing all active WebSocket connections. Proxies like Envoy don't have this behaviour, so you can reduce number of mass disconnections in your system. You can also proxy WebSocket without using ingress at all over configured WebSocket service NodePort.
\nHere comes a final part of this post. Maybe the most important one.
\nNot only mass client re-connections could create a significant load on a session backend but also a huge load on your main application database. Why? Because WebSocket applications are stateful. Clients rely on a stream of messages coming from a backend to maintain its state actual. As soon as connection dropped client tries to reconnect. In some scenarios it also wants to restore its actual state. What if client reconnected after 3 seconds? How many state updates it could miss? Nobody knows. So to make sure state is actual client tries to get it from application database. This is again a significant spike in load on your main database in massive reconnect scenario. In can be really painful with many active connections.
\nSo what I think is nice to have for scenarios where we can't afford to miss messages (like in chat-like apps for example) is having effective and performant stream of messages inside each channel. Keep this stream in fast in-memory storage. This stream can have time retention and be limited in size (think about it as a sliding window of messages). I already mentioned that Redis can do this – it's possible to keep messages in Redis List or Redis Stream data structures. Other broker solutions could give you access to such a stream inside each channel out of the box.
\nSo as soon as client reconnects it can restore its state from fast in-memory event stream without even querying your database. Actually to survive mass reconnect scenario you don't need to keep such a stream for a long time – several minutes should be enough. You can even create your own Websocket fallback implementation (like Long-Polling) utilizing event stream with limited retention.
\nHope advices given here will be useful for a reader and will help writing a more robust and more scalable real-time application backends.
\nCentrifugo server and Centrifuge library for Go language have most of the mechanics described here including the last one – message stream for topics limited by size and retention period. Both also have techniques to prevent message loss due to at most once nature of Redis PUB/SUB giving at least once delivery guarantee inside message history window size and retention period.
", "url": "https://centrifugal.dev/blog/2020/11/12/scaling-websocket", "title": "Scaling WebSocket in Go and beyond", "summary": "The post describes techniques to write scalable WebSocket servers within Go ecosystem and beyond it", "date_modified": "2020-11-12T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "websocket", "go" ] }, { "id": "https://centrifugal.dev/blog/2020/10/16/experimenting-with-quic-transport", "content_html": "UPDATE: WebTransport spec is still evolving. Most information here is not actual anymore. For example the working group has no plan to implement both QuicTransport and HTTP3-based transports – only HTTP3 based WebTransport is going to be implemented. Maybe we will publish a follow-up of this post at some point.
\nWebTransport is a new browser API offering low-latency, bidirectional, client-server messaging. If you have not heard about it before I suggest to first read a post called Experimenting with QuicTransport published recently on web.dev – it gives a nice overview to WebTransport and shows client-side code examples. Here we will concentrate on implementing server side.
\nSome key points about WebTransport spec:
\nTo help you figure out things here are links to current WebTransport specs:
\nAt moment Chrome only implements trial possibility to try out WebTransport standard and only implements WebTransport over QUIC. Developers can initialize transport with code like this:
\nconst transport = new QuicTransport('quic-transport://localhost:4433/path');
In case of HTTP/3 transport one will use URL like 'https://localhost:4433/path' in transport constructor. All WebTransport underlying transports should support instantiation over URL – that's one of the spec requirements.
I decided that this is a cool possibility to finally play with QUIC protocol and its Go implementation github.com/lucas-clemente/quic-go.
\nPlease keep in mind that all things described in this post are work in progress. WebTransport drafts, Quic-Go library, even QUIC protocol itself are subjects to change. You should not use it in production yet.
Experimenting with QuicTransport post contains links to a client example and companion Python server implementation.
\n![\"client](\"https://i.imgur.com/Hty00aG.png\")
We will use a linked client example to connect to a server that runs on localhost and uses github.com/lucas-clemente/quic-go library. To make our example work we need to open client example in Chrome, and actually, at this moment we need to install Chrome Canary. The reason behind this is that the quic-go library supports QUIC draft-29 while Chrome < 85 implements QuicTransport over draft-27. If you read this post at a time when Chrome stable 85 already released then most probably you don't need to install Canary release and just use your stable Chrome.
We also need to generate self-signed certificates since WebTransport only works with a TLS layer, and we should make Chrome trust our certificates. Let's prepare our client environment before writing a server and first install Chrome Canary.
\nGo to https://www.google.com/intl/en/chrome/canary/, download and install Chrome Canary. We will use it to open client example.
\nIf you have Chrome >= 85 then most probably you can skip this step.
Since WebTransport based on modern network transports like QUIC and HTTP/3 security is a keystone. For our experiment we will create a self-signed TLS certificate using openssl.
Make sure you have openssl installed:
$ which openssl
/usr/bin/openssl
Then run:
\nopenssl genrsa -des3 -passout pass:x -out server.pass.key 2048
openssl rsa -passin pass:x -in server.pass.key -out server.key
rm server.pass.key
openssl req -new -key server.key -out server.csr
Set localhost for Common Name when asked.
The self-signed TLS certificate generated from the server.key private key and server.csr files:
openssl x509 -req -sha256 -days 365 -in server.csr -signkey server.key -out server.crt
After these manipulations you should have server.crt and server.key files in your working directory.
To help you with process here is my console output during these steps (click to open):
\n??? example \"My console output generating self-signed certificates\"
\n$ openssl genrsa -des3 -passout pass:x -out server.pass.key 2048
Generating RSA private key, 2048 bit long modulus
...........................................................................................+++
.....................+++
e is 65537 (0x10001)
$ ls
server.pass.key
$ openssl rsa -passin pass:x -in server.pass.key -out server.key
writing RSA key
$ ls
server.key server.pass.key
$ rm server.pass.key
$ openssl req -new -key server.key -out server.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) []:RU
State or Province Name (full name) []:
Locality Name (eg, city) []:
Organization Name (eg, company) []:
Organizational Unit Name (eg, section) []:
Common Name (eg, fully qualified host name) []:localhost
Email Address []:
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
$ openssl x509 -req -sha256 -days 365 -in server.csr -signkey server.key -out server.crt
Signature ok
subject=/C=RU/CN=localhost
Getting Private key
$ ls
server.crt server.csr server.key
Now the last step. What we need to do is run Chrome Canary with some flags that will allow it to trust our self-signed certificates. I suppose there is an alternative way making Chrome trust your certificates, but I have not tried it.
\nFirst let's find out a fingerprint of our cert:
\nopenssl x509 -in server.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64
In my case base64 fingerprint was pe2P0fQwecKFMc6kz3+Y5MuVwVwEtGXyST5vJeaOO/M=, yours will be different.
Then run Chrome Canary with some additional flags that will make it trust out certs (close other Chrome Canary instances before running it):
\n$ /Applications/Google\\ Chrome\\ Canary.app/Contents/MacOS/Google\\ Chrome\\ Canary \\
--origin-to-force-quic-on=localhost:4433 \\
--ignore-certificate-errors-spki-list=pe2P0fQwecKFMc6kz3+Y5MuVwVwEtGXyST5vJeaOO/M=
This example is for MacOS, for your system see docs on how to run Chrome/Chromium with custom flags.
\nNow you can open https://googlechrome.github.io/samples/quictransport/client.html URL in started browser and click Connect button. What? Connection not established? OK, this is fine since we need to run our server :)
Maybe in future we will have libraries that are specified to work with WebTransport over QUIC or HTTP/3, but for now we should implement server manually. As said above we will use github.com/lucas-clemente/quic-go library to do this.
\nFirst, let's define a simple skeleton for our server:
\npackage main
import (
\t\"errors\"
\t\"log\"
\"github.com/lucas-clemente/quic-go\"
)
// Config for WebTransportServerQuic.
type Config struct {
\t// ListenAddr sets an address to bind server to.
\tListenAddr string
\t// TLSCertPath defines a path to .crt cert file.
\tTLSCertPath string
\t// TLSKeyPath defines a path to .key cert file
\tTLSKeyPath string
\t// AllowedOrigins represents list of allowed origins to connect from.
\tAllowedOrigins []string
}
// WebTransportServerQuic can handle WebTransport QUIC connections according
// to https://tools.ietf.org/html/draft-vvv-webtransport-quic-02.
type WebTransportServerQuic struct {
\tconfig Config
}
// NewWebTransportServerQuic creates new WebTransportServerQuic.
func NewWebTransportServerQuic(config Config) *WebTransportServerQuic {
\treturn &WebTransportServerQuic{
\t\tconfig: config,
\t}
}
// Run server.
func (s *WebTransportServerQuic) Run() error {
\treturn errors.New(\"not implemented\")
}
func main() {
\tserver := NewWebTransportServerQuic(Config{
\t\tListenAddr: \"0.0.0.0:4433\",
\t\tTLSCertPath: \"server.crt\",
\t\tTLSKeyPath: \"server.key\",
\t\tAllowedOrigins: []string{\"localhost\", \"googlechrome.github.io\"},
\t})
\tif err := server.Run(); err != nil {
\t\tlog.Fatal(err)
\t}
}
Let's concentrate on implementing Run method. We need to accept QUIC client connections. This can be done by creating quic.Listener instance and using its .Accept method to accept incoming client sessions.
// Run server.
func (s *WebTransportServerQuic) Run() error {
\tlistener, err := quic.ListenAddr(s.config.ListenAddr, s.generateTLSConfig(), nil)
\tif err != nil {
\t\treturn err
\t}
\tfor {
\t\tsess, err := listener.Accept(context.Background())
\t\tif err != nil {
\t\t\treturn err
\t\t}
\t\tlog.Printf(\"session accepted: %s\", sess.RemoteAddr().String())
\t\tgo func() {
\t\t\tdefer func() {
\t\t\t\t_ = sess.CloseWithError(0, \"bye\")
\t\t\t\tlog.Println(\"close session\")
\t\t\t}()
\t\t\ts.handleSession(sess)
\t\t}()
\t}
}
func (s *WebTransportServerQuic) handleSession(sess quic.Session) {
// Not implemented yet.
}
An interesting thing to note is that QUIC allows closing connection with specific application-level integer code and custom string reason. Just like WebSocket if you worked with it.
\nAlso note, that we are starting our Listener with TLS configuration returned by s.generateTLSConfig() method. Let's take a closer look at how this method can be implemented.
// https://tools.ietf.org/html/draft-vvv-webtransport-quic-02#section-3.1
const alpnQuicTransport = \"wq-vvv-01\"
func (s *WebTransportServerQuic) generateTLSConfig() *tls.Config {
\tcert, err := tls.LoadX509KeyPair(s.config.TLSCertPath, s.config.TLSKeyPath)
\tif err != nil {
\t\tlog.Fatal(err)
\t}
\treturn &tls.Config{
\t\tCertificates: []tls.Certificate{cert},
\t\tNextProtos: []string{alpnQuicTransport},
\t}
}
Inside generateTLSConfig we load x509 certs from cert files generated above. WebTransport uses ALPN (Application-Layer Protocol Negotiation to prevent handshakes with a server that does not support WebTransport spec. This is just a string wq-vvv-01 inside NextProtos slice of our *tls.Config.
At this moment if you run a server and open a client example in Chrome then click Connect button – you should see that connection successfully established in event log area:
![\"client](\"https://i.imgur.com/PyEr9W9.png\")
Now if you try to send data to a server nothing will happen. That's because we have not implemented reading data from session streams.
\nStreams in QUIC provide a lightweight, ordered byte-stream abstraction to an application. Streams can be unidirectional or bidirectional.
\nStreams can be short-lived, streams can also be long-lived and can last the entire duration of a connection.
\nClient example provides three possible ways to communicate with a server:
\nUnfortunately, quic-go library does not support sending UDP datagrams at this moment. To do this quic-go should implement one more draft called An Unreliable Datagram Extension to QUIC. There is already an ongoing pull request that implements it. This means that it's too early for us to experiment with unreliable UDP WebTransport client-server communication in Go. By the way, the interesting facts about UDP over QUIC are that QUIC congestion control mechanism will still apply and QUIC datagrams can support acknowledgements.
Implementing a unidirectional stream is possible with quic-go since the library supports creating and accepting unidirectional streams, but I'll leave this for a reader (though we will need accepting one unidirectional stream for parsing client indication anyway – see below).
Here we will only concentrate on implementing a server for a bidirectional case. We are in the Centrifugo blog, and this is the most interesting type of stream for me personally.
\nAccording to section-3.2 of Quic WebTransport spec in order to verify that the client's origin allowed connecting to the server, the user agent has to communicate the origin to the server. This is accomplished by sending a special message, called client indication, on stream 2, which is the first client-initiated unidirectional stream.
\nHere we will implement this. In the beginning of our session handler we will accept a unidirectional stream initiated by a client.
\nAt moment spec defines two client indication keys: Origin and Path. In our case an origin value will be https://googlechrome.github.io and path will be /counter.
Let's define some constants and structures:
\n// client indication stream can not exceed 65535 bytes in length.
// https://tools.ietf.org/html/draft-vvv-webtransport-quic-02#section-3.2
const maxClientIndicationLength = 65535
// define known client indication keys.
type clientIndicationKey int16
const (
\tclientIndicationKeyOrigin clientIndicationKey = 0
\tclientIndicationKeyPath = 1
)
// ClientIndication container.
type ClientIndication struct {
\t// Origin client indication value.
\tOrigin string
\t// Path client indication value.
\tPath string
}
Now what we should do is accept unidirectional stream inside session handler:
\nfunc (s *WebTransportServerQuic) handleSession(sess quic.Session) {
stream, err := sess.AcceptUniStream(context.Background())
if err != nil {
log.Println(err)
return
}
log.Printf(\"uni stream accepted, id: %d\", stream.StreamID())
indication, err := receiveClientIndication(stream)
if err != nil {
log.Println(err)
return
}
log.Printf(\"client indication: %+v\", indication)
if err := s.validateClientIndication(indication); err != nil {
log.Println(err)
return
}
// this method blocks.
if err := s.communicate(sess); err != nil {
log.Println(err)
}
}
func receiveClientIndication(stream quic.ReceiveStream) (ClientIndication, error) {
return ClientIndication{}, errors.New(\"not implemented yet\")
}
func (s *WebTransportServerQuic) validateClientIndication(indication ClientIndication) error {
\treturn errors.New(\"not implemented yet\")
}
func (s *WebTransportServerQuic) communicate(sess quic.Session) error {
return errors.New(\"not implemented yet\")
}
As you can see to accept a unidirectional stream with data we can use .AcceptUniStream method of quic.Session. After accepting a stream we should read client indication data from it.
According to spec it will contain a client indication in the following format:
\n0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Key (16) | Length (16) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Value (*) ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
The code below parses client indication out of a stream data, we decode key-value pairs from uni stream until an end of stream (indicated by EOF):
\nfunc receiveClientIndication(stream quic.ReceiveStream) (ClientIndication, error) {
\tvar clientIndication ClientIndication
// read no more than maxClientIndicationLength bytes.
\treader := io.LimitReader(stream, maxClientIndicationLength)
\tdone := false
\tfor {
\t\tif done {
\t\t\tbreak
\t\t}
\t\tvar key int16
\t\terr := binary.Read(reader, binary.BigEndian, &key)
\t\tif err != nil {
\t\t\tif err == io.EOF {
\t\t\t\tdone = true
\t\t\t} else {
\t\t\t\treturn clientIndication, err
\t\t\t}
\t\t}
\t\tvar valueLength int16
\t\terr = binary.Read(reader, binary.BigEndian, &valueLength)
\t\tif err != nil {
\t\t\treturn clientIndication, err
\t\t}
\t\tbuf := make([]byte, valueLength)
\t\tn, err := reader.Read(buf)
\t\tif err != nil {
\t\t\tif err == io.EOF {
// still need to process indication value.
\t\t\t\tdone = true
\t\t\t} else {
\t\t\t\treturn clientIndication, err
\t\t\t}
\t\t}
\t\tif int16(n) != valueLength {
\t\t\treturn clientIndication, errors.New(\"read less than expected\")
\t\t}
\t\tvalue := string(buf)
\t\tswitch clientIndicationKey(key) {
\t\tcase clientIndicationKeyOrigin:
\t\t\tclientIndication.Origin = value
\t\tcase clientIndicationKeyPath:
\t\t\tclientIndication.Path = value
\t\tdefault:
\t\t\tlog.Printf(\"skip unknown client indication key: %d: %s\", key, value)
\t\t}
\t}
\treturn clientIndication, nil
}
We also validate Origin inside validateClientIndication method of our server:
var errBadOrigin = errors.New(\"bad origin\")
func (s *WebTransportServerQuic) validateClientIndication(indication ClientIndication) error {
\tu, err := url.Parse(indication.Origin)
\tif err != nil {
\t\treturn errBadOrigin
\t}
\tif !stringInSlice(u.Host, s.config.AllowedOrigins) {
\t\treturn errBadOrigin
\t}
\treturn nil
}
func stringInSlice(a string, list []string) bool {
\tfor _, b := range list {
\t\tif b == a {
\t\t\treturn true
\t\t}
\t}
\treturn false
}
Do you have stringInSlice function in every Go project? I do :)
The final part here is accepting a bidirectional stream from a client, reading it, and sending responses back. Here we will just echo everything a client sends to a server back to a client. You can implement whatever bidirectional communication you want actually.
\nVery similar to unidirectional case we can call .AcceptStream method of session to accept a bidirectional stream.
func (s *WebTransportServerQuic) communicate(sess quic.Session) error {
\tfor {
\t\tstream, err := sess.AcceptStream(context.Background())
\t\tif err != nil {
\t\t\treturn err
\t\t}
\t\tlog.Printf(\"stream accepted: %d\", stream.StreamID())
\t\tif _, err := io.Copy(stream, stream); err != nil {
\t\t\treturn err
\t\t}
\t}
}
When you press Send button in client example it creates a bidirectional stream, sends data to it, then closes stream. Thus our code is sufficient. For a more complex communication that involves many concurrent streams you will have to write a more complex code that allows working with streams concurrently on server side.
![\"client](\"https://i.imgur.com/5299Vr4.png\")
Full server code can be found in a Gist. Again – this is a toy example based on things that all work in progress.
\nWebTransport is an interesting technology that can open new possibilities in modern Web development. At this moment it's possible to play with it using QUIC transport – here we looked at how one can do that. Though we still have to wait a bit until all these things will be suitable for production usage.
\nAlso, even when ready we will still have to think about WebTransport fallback options – since wide adoption of browsers that support some new technology and infrastructure takes time. Actually WebTransport spec authors consider fallback options in design. This was mentioned in IETF slides (PDF, 2.6MB), but I have not found any additional information beyond that.
\nPersonally, I think the most exciting thing about WebTransport is the possibility to exchange UDP datagrams, which can help a lot to in-browser gaming. Unfortunately, we can't test it at this moment with Go (but it's already possible using Python as server as shown in the example).
\nWebTransport could be a nice candidate for a new Centrifugo transport next to WebSocket and SockJS – time will show.
", "url": "https://centrifugal.dev/blog/2020/10/16/experimenting-with-quic-transport", "title": "Experimenting with QUIC and WebTransport", "summary": "Here we experiment with QUIC and WebTransport in Chrome", "date_modified": "2020-10-16T00:00:00.000Z", "author": { "name": "Alexander Emelin" }, "tags": [ "quic", "webtransport", "go" ] }, { "id": "https://centrifugal.dev/blog/2020/02/10/million-connections-with-centrifugo", "content_html": "In order to get an understanding about possible hardware requirements for reasonably massive Centrifugo setup we made a test stand inside Kubernetes.
\nOur goal was to run server based on Centrifuge library (the core of Centrifugo server) with one million WebSocket connections and send many messages to connected clients. While sending many messages we have been looking at delivery time latency. In fact we will see that about 30 million messages per minute (500k messages per second) will be delivered to connected clients and latency won't be larger than 200ms in 99 percentile.
\nServer nodes have been run on machines with the following configuration:
\nSome sysctl values:
fs.file-max = 3276750
fs.nr_open = 1048576
net.ipv4.tcp_mem = 3086496\t4115330\t6172992
net.ipv4.tcp_rmem = 8192\t8388608\t16777216
net.ipv4.tcp_wmem = 4096\t4194394\t16777216
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
Kubernetes used these machines as its nodes.
\nWe started 20 Centrifuge-based server pods. Our clients connected to server pods using Centrifuge Protobuf protocol. To scale horizontally we used Redis Engine and sharded it to 5 different Redis instances (each Redis instance consumes 1 CPU max).
\nTo achieve many client connections we used 100 Kubernetes pods each generating about 10k client connections to server.
\nHere are some numbers we achieved:
\nAnd here are some numbers about final resource usage on server side (we don't actually interested in client side resource usage here):
\nThe picture that demonstrates experiment (better to open image in new tab):
\n
This also demonstrates that to handle one million of WebSocket connections without many messages sent to clients you need about 10 CPU total for server nodes and about 5% of CPU on each of Redis instances. In this case CPU mostly spent on connect/disconnect flow, ping/pong frames, subscriptions to channels.
\nIf we enable history and history message recovery features we see an increased Redis CPU usage: 64% instead of 32% on the same workload. Other resources usage is pretty the same.
\nThe results mean that one can theoretically achieve the comparable numbers on single modern server machine. But numbers can vary a lot in case of different load scenarios. In this benchmark we looked at basic use case where we only connect many clients and send Publications to them. There are many features in Centrifuge library and in Centrifugo not covered by this artificial experiment. Also note that though benchmark was made for Centrifuge library for Centrifugo you can expect similar results.
\nRead and write buffer sizes of websocket connections were set to 512 kb on server side (sizes of buffers affect memory usage), with Centrifugo this means that to reproduce the same configuration you need to set (config updated for Centrifugo v6+):
\n{
\"websocket\": {
\"read_buffer_size\": 512,
\"write_buffer_size\": 512
}
}