Systems10 min read
Fix our websocket server, or adopt one?
Our in-house websocket layer was struggling with live perps order books. The bounded fixes looked cheap, adopting Centrifugo looked expensive, and the decision came down to which one I'd still trust in six months.
From Print.World · Aug 2025 – Present
When we started streaming live perpetual futures order books to the terminal, our websocket layer started to creak. This post is about the decision I had to make there: patch the server we already had, or put a dedicated real-time server next to it. It's the kind of decision that doesn't have a right answer, only a defensible one, so I want to show the reasoning, not just the outcome.
The problem
Print.World's terminal gets live data over an in-house websocket service. It had served us well for prices and trades. Order books are different: they update many times a second, every viewer of a market needs them, and a client that misses an update has a wrong book until it gets a fresh snapshot.
Looking at how the existing server handled that, a few structural issues stood out:
- Every instance received every message. Fan-out ran over a single Redis pub/sub channel, so each server instance processed traffic for markets none of its clients were watching.
- Work per socket, not per message. Each outgoing message was serialized and compressed once per connected socket, so the cost scaled with viewers instead of with updates.
- No backpressure and no recovery. A slow client just accumulated a growing send buffer, and a client that reconnected had no way to ask for what it missed.
- The wrong scaling signal. The service autoscaled on load-balancer requests per target. For a websocket that's one request at connect time, so the signal tracked new connections, not messages or bytes, which is where the cost actually was.
Here's the load the perps app implied. The market universe was 698 markets. A 25-level book with a per-venue breakdown is 10 to 20 KB of JSON, and the busiest books, with every venue's depth folded in, are closer to 50 KB. Books publish every 250ms, so four frames a second per symbol, and the live set is capped at 10 streamed symbols platform-wide.
We'd already had a warning shot: during one incident, pub/sub ingress jumped from about 13 to 60 MB/s and Redis closed the subscriber connection. Perps books would make that traffic heavier, not lighter.
A back-of-envelope with those inputs shows where it breaks first. Ten symbols at four 50 KB frames a second is about 2 MB/s of book traffic published. With one shared channel, every instance receives all of it, whether or not any of its clients watch those markets, so ingress grows with instances times publishes. Then the per-socket work: each frame is serialized and deflated once per watching socket, so 1,000 viewers of one book means about 4,000 compressions a second, around 200 MB/s of JSON fed through deflate, for a single market. Neither number is a measurement; they're the arithmetic that says the bottlenecks arrive in order: Redis output buffers first, then server CPU, then client send buffers.
Option one: four bounded fixes
The cheap-looking option was to fix the existing server in place. I wrote down exactly what that would take:
- per-channel subscriptions, so instances only receive markets their clients want
- compress once per message, not once per socket
- a ceiling on each socket's send buffer, so a slow client gets dropped instead of hurting everyone
- sequenced deltas, so clients can detect a gap and recover
Each of those is a reasonable change. Together, they touch every real-time feature on the platform, not just perps. Only the fourth is perps-specific; the first three change code every message on the platform flows through, and in the design doc I sized each of those as a multi-week change. And when I listed them out, I noticed something uncomfortable: I was describing, piece by piece, a subset of what a dedicated real-time server already does. Even all four would still have left the server without recovery history, per-channel metrics, or a delta on the wire.
Option two: Centrifugo, next to what we have
Centrifugo is an open-source real-time messaging server. It already does channel subscriptions, history and recovery with offsets and epochs, and delta compression: sending a patch against the last message instead of the whole thing.
Each of the four fixes maps to a mechanism it already has, as configuration rather than code:
- Subscribed fan-out. Each node subscribes to a market's Redis channel only while it has a local subscriber for that market, so a BTC publish reaches only the nodes that carry a BTC watcher.
- Recovery. Every channel keeps a short history stamped with an offset (a per-channel sequence number) and an epoch (which changes if that history is lost). A reconnecting client sends its last offset and either receives exactly the publications it missed, or gets
recovered: falseand knows to take a fresh snapshot. It never silently resumes on a hole. - Small frames. With Fossil delta compression, the server sends the diff between consecutive publications on the same channel. The publisher keeps sending full snapshots; the wire carries patches.
- Backpressure. Each client has a bounded write queue, and a client that can't keep up is disconnected with a distinct code instead of growing a buffer on the server.
Its published figures were comfortably past what we needed: about 500,000 messages a second delivered with p99 under 200ms in their benchmark, and about 30 KB of memory per connection, so 10,000 connections is on the order of 300 MB.
The costs were real, too. There's no license fee, but it's a second service to deploy, monitor, and understand, and the team would have two real-time paths for a while. It also meant a second client library in the frontend and a token-minting endpoint in our API, so the browser could carry its existing login to a server that doesn't speak our auth.
| Fix in place | Adopt Centrifugo for perps | |
|---|---|---|
| Blast radius | Every real-time feature | Perps namespace only |
| Recovery after reconnect | Build it | Built in (history, offsets, epochs) |
| Delta compression | Build it | Built in |
| New operational surface | None | One more service |
| Risk of subtle regressions | High, shared code | Low, isolated |
The deciding factor wasn't performance. It was blast radius. Fixing the shared server meant risking regressions in features that had nothing to do with perps, in code every real-time message flows through. Adding a second server meant more to run, but a failure in it could only hurt the thing I was building.
Getting the details right
A few pieces I'm happiest with:
- Gaps are errors, not warnings. Venue order-book feeds are sequenced. If a delta doesn't chain onto the last one (the next sequence number, or on some venues a "previous update" field that must match the book's last timestamp), the book is marked stale and the client re-fetches a full snapshot. Applying an update past a gap would silently serve a wrong book, which is worse than briefly serving none.
- One subscription per channel per client. On the terminal side, a reference-counted client means ten components watching the same market share one subscription instead of opening ten.
- Delta compression on the wire. Clients ask for deltas, so after the first full book, each message is a small patch.
- Recovery matched to what each channel means. Not every stream needs the same history:
| Channel | History kept | On reconnect |
|---|---|---|
| Book, ticker, live candle | the latest publication | the newest state; nothing older matters |
| Trade tape | last 100 publications, 5 minutes | every print missed, in order |
| A user's account | the latest publication | newest state, and only that user may subscribe |
A book is a state: if you missed three deltas, the right answer is the current book, not a replay. A tape is a log: a missing print is a hole a trader can see. The tape publishes at most every 250ms, so 100 publications cover at least 25 seconds of a market printing continuously, and longer for quieter ones. A reconnect inside that window replays the gap exactly; past it, the client is told recovered: false and re-seeds from history instead of pretending.
- Private channels without a proxy. The account channel is user-limited: only a connection whose token names that user id can subscribe. Our API mints the token (HMAC-signed, 15-minute lifetime) from the user's existing login, so authorization stays in a token we sign and there is no per-subscribe callback to our API.
- A slow push server can't slow the publisher. The publish client never throws. Each request has a 2-second timeout, and a transport failure opens a 5-second circuit, so a hung Centrifugo costs one timeout and then a cheap skip, instead of stretching every 250ms tick by two seconds.
- One request per tick, not one per frame. Every publication queued within 10ms goes out as one batch request, with one request in flight at a time so order is preserved. A tick's ticker, tape and four candle frames per symbol become one call instead of six per symbol, so 60 calls a tick at the 10-symbol cap become one.
Show data
| mode | KB/s |
|---|---|
| Full book every update | 132 |
| Delta compression | 6.40 |
That's about a 20x reduction in bandwidth for the busiest book, measured locally. I'm careful to say "locally": production rollout was still pending its infrastructure when I wrote the design doc, and the doc says so.
Why it's that large: 6.4 / 132 means roughly 95% of each book frame is unchanged from the one before it. That's what a book looks like at 250ms resolution. Most levels don't move between two snapshots a quarter of a second apart; a few prices and sizes near the top do. The gain is smaller everywhere that isn't true. The trade tape was already a delta (each frame carries only new prints), so it gains little. The very first frame after subscribing is always a full snapshot. And a fast-moving book in a volatile minute changes more levels per tick, so its ratio will be worse than this quiet local run.
Bandwidth is also the cost line. The perps cost model puts a typical viewer, with book, tape, ticker and candles open, at about 220 KB/s without deltas and about 20 KB/s with them. At the public AWS internet egress price of about $0.09 per GB, that's roughly 0.79 GB and 7 cents per viewer-hour without deltas, against about 0.07 GB and under a cent with them. Browser egress is most of what the app costs to run, so this is the setting that moves the bill.
Where this design stops
A few limits I'd want any reviewer to know about:
- The streamed set is capped at 10 symbols. That cap comes from the venues, not the push layer: venue websockets and REST budgets are counted per egress IP. Markets outside the set are priced from batch REST reads. Lifting it means more egress IPs on the ingest side, not a bigger Centrifugo.
- Redis is still in the path. Centrifugo uses Redis as its broker and history store, so at much higher fan-out the broker is the next thing to size, and it should be a Redis whose output buffers aren't shared with the rest of the app's pub/sub. Centrifugo's published figure of about a million publications a second through one Redis is far above ten symbols at a few frames a second, so this is a 10x-and-beyond concern, not a today one.
- Ordering is per channel only. Offsets order publications within one channel. A book frame and a tape frame for the same market can arrive in either order, so anything that needs both has to treat them as independent streams.
- Initial state got cheaper, then I trimmed the writes behind it. Centrifugo's cache recovery serves a new subscriber the latest book, so the Redis copy kept for the old server's initial data no longer needs to be written on every tick. Storing it four times a second was about 630 KB/s of Redis ingress for three symbols; storing it once a second cuts that by about 4x, to roughly 160 KB/s.
What I took from it
List the fixes before you choose them. Writing the four bounded fixes out explicitly is what showed me I'd be rebuilding a real-time server badly. Without the list, "just patch it" would have felt obviously cheaper.
Optimize for blast radius when you're unsure. I couldn't prove Centrifugo would be faster to ship. I could prove a mistake in it couldn't break trades, alerts, or anything else on the platform.
Keep the escape hatch. The live switch back to the old server cost almost nothing to build and turned a one-way door into a two-way one. That made the decision much easier to get approved, and much easier to sleep on.
I used the same pattern earlier on BRDG, where every real-time message also has a polling fallback. Real-time is a performance feature; correctness should never depend on it.