Written October 2026, looking back at Systems8 min read
The terminal and the engine
Joining Print.World to work on the Rust trade backend under a crypto trading terminal, and the first problems that taught me how it really behaves.
From Print.World · Aug 2025 – Present
I joined Print.World as a software engineer in August 2025. It's a crypto trading terminal, remote from Toronto, and I'm doing it alongside school. The terminal is what users see. I work on the engine underneath it: quotes, routing, and execution, mostly in the Rust backend that builds and sends trades on Solana.
These are the first few problems I took on there, because each one taught me something about how the system actually behaves rather than how I assumed it would.
Building swaps across many DEXs
A trade on-chain isn't one API call. You pick a venue, read its pool state, derive the right accounts, and build an instruction in that venue's exact format. The backend supports swap building across roughly eight DEX families, and each has its own account layout and quirks.
Most of the work is careful plumbing. The fun part is that "careful" has a very literal meaning here: get one account wrong and the transaction fails on-chain, after you've already told the user it's on its way.
A cache that verifies itself
That leads to my favourite design on the backend so far. We cache pool data so we don't hit RPC on every quote. Some of the cached fields are used to derive on-chain addresses. If one of those values is plausible but wrong, you build a transaction that looks completely valid, passes every local check, and then reverts on-chain.
That's the worst kind of bug: no error until it costs the user something.
The fields that matter are the ones a program re-derives an address from. Solana programs often check that an account you pass is the program-derived address (PDA) computed from some seed, such as a pool's creator. If the cached creator is wrong, the derived account is wrong, and the program rejects the whole transaction with a seed-constraint error. The same shape applies to a mint's token program (two token programs exist, and the associated token account derives differently under each) and to a few layout flags that change which accounts a venue expects.
Making every read go to chain would defeat the point of the cache. So instead, after every cache write, a background task re-reads that account from chain and compares. The rules for that task mattered more than the idea:
- it never blocks the caller; the trade path doesn't wait for verification
- it only repairs when the on-chain value actually differs
- if the RPC call errors, it writes nothing, because a failed read is not evidence the cache is wrong
- it's debounced per key, so a burst of writes to the same entry costs one verification, not one RPC call each
The numbers behind those rules: the debounce window is 30 seconds per (field kind, key), the verifier waits 500ms after the write before reading so it doesn't race its own write or read a slot from before it, and the debounce map itself is soft-capped at 4,096 entries and swept when it grows past that. The window also bounds cost with simple arithmetic. However hot a pool is, it costs at most one verification read per 30 seconds, so N distinct keys being written cost at most N / 30 reads a second: a thousand hot pools is about 33 reads a second, whatever the trade rate.
The window also closes a loop that would otherwise be easy to miss. A repair is itself a cache write through the same setter, so it re-enters the verifier. Because the debounce slot is claimed before the background task is spawned, and 30 seconds is far longer than the half-second settle plus one RPC call that a repair takes, the re-entry is a guaranteed no-op. A repair can never trigger another verification.
It only compares fields that are static once an account exists (a pool's mints, its creator, a mint's owning program), never fields that move with every trade, like reserves. That's what makes a mismatch meaningful: a static field that disagrees with chain is a bad write, not a race with the market.
One subtlety I'd tighten: a slow verification could read the chain at an older slot than the value already in the cache. Every RPC response carries the slot it was read at, so a repair should only apply when its read is at least as new as the cached value. Otherwise the verifier itself could make the cache wrong. The reads use processed commitment, the freshest and least final view a node has, which keeps that window small but adds one more case to the same rule: a processed read can, rarely, come from a fork that gets dropped.
Making a gap measurable
We consume a live stream of chain updates. Streams disconnect; that's normal. The problem was that, in the version we ran, the stream couldn't resume from a specific slot. After a reconnect, there was a window of updates we'd simply never seen, and nothing told us how big it was.
I couldn't close the gap with what the stream supported at that version. What I could do was make it measurable. On reconnect, we compare the last slot we processed with the first slot we receive and record an upper bound on how many slots we might have missed. That number feeds alerting. A small gap is noise. A large one means the cache may be stale and someone should look.
The arithmetic is deliberately simple. If the last slot before the drop was L and the first slot after the resubscribe is R, the gap is R − L − 1, the slots strictly between two slots we did see. It's an upper bound rather than a count, because Solana leaders sometimes skip their slot entirely, so some of those slot numbers never had a block to miss. A slot is about 400ms, so the bound converts directly into time: a reconnect that resumes 25 slots later lost at most about ten seconds of chain. The last-seen slot is kept across reconnect sessions, the check runs once per session on its first transaction, and a first connect (nothing to compare against) or an out-of-order resume reports zero instead of a nonsense number.
It's not a fix, and I'm upfront about that internally. But "we lose some updates sometimes" and "we lost at most N slots at this time" are very different statements to operate against.
Where did the milliseconds go?
Users notice when a trade feels slow, and "it feels slow" isn't actionable. So I added a per-trade latency breakdown, for both the quote and the submit path. Every RPC call and cache call gets timed, cache misses are recorded, and we note where the blockhash came from. The result reads roughly like:
quote: cache_pool 0.x ms (hit) | rpc_account N ms (miss) | build N ms
submit: blockhash (cached) | sign N ms | send N ms
Before that, latency debates were opinions. After, you could point at the line that was slow.
A few details make the breakdown trustworthy rather than just present:
- Positioned segments, not just durations. Each leg returns an ordered timeline of
{name, at_ms, ms}, whereat_msis the offset from the leg's start. Durations alone can't tell you whether two calls overlapped; offsets can. - Sums are labelled as sums. The RPC and Redis totals add up every call's time, and on the hot path those calls run in parallel, so
rpc_mscan exceed the leg's wall-clocktotal_ms. That's correct, and it's documented, because the first time someone sees it they'll assume a bug. - Absolute anchors for the network. Each leg also carries its start and end as epoch milliseconds, so the client can subtract its own send and receive times to get time on the wire in each direction. That subtraction is only as good as the two clocks agree, so it's a rough number; the server-side segments are exact.
Freshness versus cost
A lot of the backend is balancing how fresh data needs to be against how much RPC you're willing to spend. Two examples:
- the recent blockhash is polled into a cache in the background rather than fetched per trade, so submit doesn't pay for that round-trip
- negative results, like "this account doesn't exist", get a shorter TTL than positive ones, because they're the ones most likely to change soon and the most damaging to keep wrong
Here's why the blockhash is safe to cache. A transaction names a recent blockhash, and the network accepts it for about 150 blocks, on the order of a minute. A cached hash a second old has used up about two or three slots of that budget, so nothing is lost by not fetching it per trade. The cache takes new hashes from the same block stream we already consume, keeps the last ten, and falls back to polling RPC every 400ms (one slot) only when the stream has been silent for 1.2 seconds, about three slots.
The negative TTL is easiest to see on one cache. Tick-array discovery for concentrated-liquidity pools is cached for an hour when it finds arrays, because they're created once and rarely change. An empty result is cached for 45 seconds. The asymmetry is the point: a stale positive costs a slightly worse route, but a stale "this pool has no liquidity" makes a pool that just went live look dead for up to an hour. At 45 seconds, a newly liquid pool recovers within a minute, while a genuinely empty pool still costs at most one extended search every 45 seconds instead of one per quote.
Neither is clever. Both came from looking at the breakdown and asking why a particular number was there.
What I've taken from the first month
Every fix above started with measuring, not guessing. The cache bug was invisible until I asked what happens when a value is plausible but wrong. The stream gap was invisible until I tried to quantify it. The latency complaints went nowhere until each trade could explain its own milliseconds.
The latency breakdown made every later conversation shorter, and it's the first thing I build on any new path now. Later work at Print took me into perps aggregation and bridging, and the same habit carried over.