Written October 2026, looking back at Systems7 min read
Taking the lock off the hot path
Every swap our Rust market-data processor decoded went through one global lock, and a cache hit did O(n) work while holding it. Replacing it took cache lookups from about 310k a second, on any number of cores, to 91 million on ten.
From Print.World · Aug 2025 – Present
At Print.World, the trade path for Solana starts with a Rust service I spend a lot of time in: the DEX processor. It reads a gRPC stream of chain transactions, recognizes swaps across the venues we support, keeps caches of pool state up to date, and publishes what it learns to Kafka and Redis for the rest of the backend. If it falls behind, every price and every quote downstream is a little bit wrong.
In January I wanted to know how much headroom it actually had. Not "it seems fine in prod", but a number I could defend. Reading the hot path for that turned up a lock that every swap, on every DEX we decode, had to pass through, and that put a ceiling on the whole process no matter how many cores it ran on.
Measure the pipeline, then isolate the change
The rule I keep coming back to, since my first internship, is that you don't optimize what you can't measure repeatably. I started with a load harness for the publish side of the pipeline: it drives the Kafka producer, configured the way production configures it, as fast as it will go for 65 seconds, records every latency into an HdrHistogram (p50 through p999 and max), and commits each run as a timestamped JSON file so results are artifacts anyone can diff. That tells me what the publish path can sustain on a given machine. It doesn't run the decoders, so it can't score a change to them.
For a concurrency change, an end-to-end number is the wrong instrument anyway: it mixes the change with everything else on the path and with whatever the machine was doing that minute. So the test for this change was narrower and stricter. I took the cache from before and after the change, copied verbatim into a standalone benchmark, filled it the way production fills it, and hammered it from 1 to 10 threads on the same machine, with the lookup the only thing being timed.
What every swap was waiting on
Every swap the processor decodes needs the decimals of the tokens involved, to turn raw on-chain amounts into prices. The decoders for all four DEX families we supported then (Pump.fun, Raydium, Orca, and Meteora) fetched them through one shared helper. Reading that helper and the publish path beside it, three things stood out:
- A global
Mutexaround a cache. One of the caches was aHashMapplus aVecDequeused as an LRU, behind a single mutex. Every lookup took the lock, and every cache hit calledretainon the deque to move the key to the front, which is O(n) in the size of the cache. So the most common operation, a hit, was also the most expensive one, and it held a lock everyone needed while doing it. Concretely, the cap defaulted to 1,000 entries, so a hit compared the key against up to 1,000 mint-address strings, then allocated a freshStringto push back onto the deque, all inside the critical section. - A
RwLock<HashMap>of subscriber sockets. Reads dominated, but writers still stalled every reader, and the publish path touched it on every message. It was worse than "read-mostly" suggests: the get-or-create helper took the write lock unconditionally, before checking whether the channel already existed, so even the common case of an existing subscriber serialized behind a write. Tokio'sRwLockis write-preferring, so one queued writer also blocks every reader that arrives after it. SeqCststores on the publish path. Shared prices were published with sequentially consistent stores. Readers only needed to see a complete value once it was written, which aReleasestore paired withAcquireloads guarantees. On x86 that's a real difference for stores: aSeqCststore compiles to anXCHG, which acts as a full barrier, while aReleasestore is a plainMOV. On ARM (AArch64) it's no difference at all: both compile to the sameSTLRinstruction, so this change only pays on x86 hosts. It's still the right ordering to write:Release/Acquireis what the code needs, and it says so to the next reader.
The change
None of the fixes are exotic, which is sort of the point:
- the mutex-guarded cache and the socket map both became
DashMap, a concurrent hash map that shards its keys across many internal locks, so threads only contend when they touch the same shard. By default it creates four shards per available CPU, rounded up to a power of two: 64 shards on a 10-CPU machine, so two threads touching random keys collide about 1 time in 64 instead of every time - the socket map's get-or-create became a shared read first, and only on a miss an
entry().or_insert_with(), which locks the one shard that key lives in - the O(n) LRU bookkeeping on every hit went away. The cache holds token decimals, which never change for a given token, so exact recency doesn't matter: when it reaches its configured cap (the default went from 1,000 to 10,000 entries) it evicts about a tenth of its entries in one batch, keeping memory bounded without per-hit bookkeeping. The price of dropping LRU is that the batch evicts arbitrary entries, hot ones included; for a value that costs one Redis read to refetch, that's a better trade than paying O(n) on every hit
- price stores on the publish path moved from
SeqCsttoRelease, withAcquireon the read side - the per-token broadcast channel shrank from 1,000,000 slots to 100,000. Tokio's broadcast channel allocates every slot up front (rounded up to a power of two), and each slot holding a
Stringis on the order of 40 to 50 bytes, so that's roughly 50 MB per token channel down to roughly 6 MB. A memory fix, not a latency one, but it shipped in the same change
What it bought
The isolated benchmark ran on a 10-core Apple M4, with the cache full at the old default of 1,000 token addresses and every lookup a hit, which is the steady state in production: the set of actively traded tokens is small and hot.
Show data
| Threads | Old cache |
|---|---|
| 1 | 538,178 |
| 2 | 380,229 |
| 4 | 335,222 |
| 8 | 310,659 |
| 10 | 312,622 |
That curve is the whole story. On one thread the old cache did about 538,000 lookups a second, roughly 1.9 microseconds each, most of it the retain scan. Add threads and the total fell, settling around 310,000 a second, because only one thread can hold the lock and the rest pay to wait for it. At one or two decimals lookups per swap, that caps the entire process at a few hundred thousand swaps a second, whatever hardware it runs on.
| Threads | Old: lookups/s | Old: p99 | New: lookups/s | New: p99 |
|---|---|---|---|---|
| 1 | 538k | 5.1 µs | 38.0M | 83 ns |
| 4 | 335k | 187 µs | 82.4M | 167 ns |
| 10 | 313k | 550 µs | 91.1M | 250 ns |
The new cache scales the other way: 38 million lookups a second on one thread and 91 million on ten, 290 times the old cache at the same thread count. The tail is the bigger change. At ten threads the old p99 was 550 microseconds, because a thread could queue behind several others each doing a 1,000-entry scan; the new p99 is 250 nanoseconds. A lookup went from a cost the decoders had to budget for to one that doesn't show up.
Two honest limits. This measures the cache, not the processor end to end, so it shows the ceiling that was removed, not how much faster a production day became; the processor still does real work per swap after the lookup. And the SeqCst change wouldn't show on this machine at all: AArch64 compiles both orderings to the same instruction, so it only pays on x86 hosts.
What I took from it
- Throughput that falls as you add cores is a serialization point. Plotting one thing against thread count tells you what kind of problem you have before you open a profiler.
- The expensive operation is often the common one. A cache hit was doing more work than a miss, under a lock. Nobody designed it that way; it accreted.
- Isolate the change you're scoring. An end-to-end run is right for sizing a pipeline and wrong for crediting one change. Copy the before and after code into the same benchmark, vary one thing, and commit it so the next person can rerun it instead of trusting me.
The natural next step is wiring both into CI: the cache benchmark with a threshold on multi-threaded throughput, so a change that reintroduces a global lock fails review, and the pipeline harness with a loose threshold on p99.