Systems8 min read
Designing for the retry
Any code that moves money will eventually run twice. The design rules I use so that "twice" is always harmless, with the state machines and diagrams behind them.
From Print.World · Aug 2025 – Present
Every system that moves money has retries in it: a client retries a request, a job runner retries a task, a network library resends a packet. Retries are good. They're also the most common way a correct system does something twice.
So when I design a money path, I start with one question: what happens if this runs again after the money already moved? This post is how I answer it.
The shape of the problem
The dangerous pattern isn't a crash. It's a sequence where every individual piece is reasonable:
Retrying a failed order is reasonable. Looking up the trade afterwards is reasonable. A timeout on a slow query is reasonable. The problem is the composition: a retry boundary drawn around work that already partly succeeded, and an error from a non-critical step allowed to cross it. It's a well-known shape in trading systems, and it's the class of bug I design against from the first line.
Six design rules
1. Draw the retry boundary at the side effect. Retry the thing that hasn't happened yet. Once money has moved, everything after it is bookkeeping, and bookkeeping failures get logged and reconciled, never retried by re-running the trade.
2. "I don't know" is a state. On the perps work, a close or a take-profit/stop-loss order that fails without a clear answer from the venue isn't treated as failed. It's stored as UNKNOWN, and a retry replays the stored outcome instead of acting again. An authoritative rejection from the venue itself, on the other hand, means the order was never accepted, so it's safe to try again. Reconciliation works because every order carries a client order ID, so the system can ask the venue "did you get order X?" instead of guessing.
venue says ok -> PLACED (done)
venue says refused -> REJECTED (nothing sent, safe to retry)
no clear answer -> UNKNOWN (reconcile, never resend blindly)
"No clear answer" has a precise definition, because it's the line between a safe retry and a double fill: a timeout after the request left, a response that doesn't parse, or a connection dropped mid-call. Each of those means the venue may have acted. The mechanics that make replay work:
- The client order ID is the idempotency key. The app generates it once per order and reuses it on every retry. A retry with the same ID replays the stored reply, flagged as a replay, for 24 hours. An ID already spent on a different action, or reused with different parameters (venue, symbol, side, size, time-in-force), is refused with a conflict and nothing is sent.
- Intent is written before the side effect. For a multi-leg order, each leg is marked
sendingin the idempotency record before it goes out. If the reply is lost, a retry finds those legs and answers themUNKNOWN, and only places legs that were never sent. Without that write-ahead mark, a retry can't tell "never sent" from "sent, reply lost", which is exactly the distinction that matters. - Concurrent retries don't race. An order holds a lock for its whole run, and a second attempt with the same ID while the first is in flight gets "order in flight" instead of a second send. The record is only written while the lock is still held, so an attempt that takes over after a lock expiry owns the record.
- No store, no order. If the idempotency store can't be reached, the request fails closed with "idempotency unavailable". Placing an order without the ability to deduplicate it is the one thing this design never does.
The 24-hour window is sized against the retry horizon, not the order's lifetime. A client retry, a job runner retry or a user pressing the button again all happen within seconds to minutes; a day leaves several orders of magnitude of margin. After the window, an ID whose legs are already in the order ledger is still refused rather than reused, so expiry of the fast store never turns into a second order.
3. Some things expire, and must stay expired. For cross-chain deposits to a single-use address, I made sure the system never re-broadcasts a deposit after its deadline. Re-sending late opens a window where two deposits can land.
4. Resending the same bytes is safe. Rebuilding is not. On Solana, rather than rely on SDK helpers that resend while they poll (which can report a trade that already landed as a failure), I send once, poll confirmation myself, and allow at most one guarded resend of the identical signed transaction. Identical bytes carry an identical signature, so they can only land once. Building a fresh transaction to "retry" is how you buy twice. I wrote more about this in nights and weekends on Solana.
In the submit path where I made this change, the schedule is small enough to state in full. Send once. A send that times out is treated as "may have landed" and goes straight to confirmation, not to an error. An "already processed" answer from the node means it landed, so it's treated as success. Then poll the signature every 500ms, up to 20 times, a 10-second window. If the signature is still absent from the status index after two empty polls (about a second), resend the identical bytes once, then keep polling. Anything else the node says about the send, such as a program error in preflight, is a real failure and is reported with the simulated logs.
The resend is safe for a reason worth spelling out: the network deduplicates by signature, so the worst a duplicate copy can do is be rejected as already processed. The SDK helper I replaced resent while it polled, and when its first send landed before the status index caught up, its own resend came back "already processed" and it reported a successful trade as a failed submit. The user would then retry, and that retry is a new transaction.
The same rule holds on the bridging side, where it's stricter still. A bridge broadcast that times out or returns something unreadable is recorded as UNKNOWN and is never re-broadcast at all: a re-signed EVM transaction takes a fresh nonce, so it's a second transfer, not a duplicate. Only the status poller resolves an UNKNOWN, and the API answers it with a 202 rather than a 200, so the caller can tell "finished, here's the status" from "stopped with an outcome I can't read, don't retry".
5. Status is a version. On BRDG, a transfer row's status acts as its version number, and a writer only updates the row if the status is still what it read. Two pollers racing on the same transfer can't overwrite each other: a slow poll can never stamp a received amount onto a row that has already moved to REFUNDED. This works because those transitions don't revisit a status; where one can (an UNKNOWN that goes back to pending), a separate version number that only increases is the safe design.
A weaker form of the same guard is often enough, and it's what the bridge executor in our trading backend uses for its status writes: every automatic update carries AND status <> ALL(<terminal statuses>). That doesn't stop two non-terminal writers from interleaving, but it makes terminal states absorbing. Once a row is filled, refunded or failed, no late poll can move it again, and that's the property money actually depends on.
6. Enforce "once" in two places. A check-then-act sequence ("is there already an execution for this quote? no? then sign") is only safe while nobody else is between the check and the act. So the bridge executor enforces one execution per quote decision twice:
- a per-decision lock is taken before the check and released in
finally; a second caller gets an immediate "execution in progress" conflict instead of queueing behind the first - underneath it, a partial unique index on
(decision_id) WHERE status <> 'FAILED'rejects a racing insert at the database, before anything is broadcast
The lock handles the common case cheaply; the index is the guarantee if the lock ever expires mid-call or is bypassed. FAILED is excluded on purpose: a definite failure left nothing on chain, so executing that decision again is asking for the transfer the user never got. The execution row, with the whole plan and no transaction hashes yet, is written before the first broadcast, and each hash is written the moment it's known, so a crash between steps leaves a row the poller can find.
Reconciliation needs a deadline too, or UNKNOWN becomes a place money goes to be forgotten. The bridge poller sweeps open transfers every 10 seconds, up to 500 rows per sweep, oldest-waiting first, so the rows nearest their deadline are never the ones left out. A transfer gets 20 times its quoted ETA to finish, floored at 30 minutes and capped at 24 hours, so a missing or absurd ETA still gives a sane deadline. Past it, the row escalates to a state that says a human settles it. No kill switch stops this sweep: switching a venue off stops new transfers to it, never the tracking of money already in flight.
What I took from it
The scariest bugs in trading systems aren't crashes. Crashes are loud. These are quiet: the system does exactly what each piece was told to do, and the sum of it is a second sell nobody asked for.
So I don't ask "what happens if this fails?" I ask "what happens if this fails after the money moved?" That one question shapes more of my code than any library.