← All writing

Written October 2026, looking back at Research8 min read

The missing 300ms

My real-time multimodal model for scoring a social feed was lagging by a few hundred milliseconds, and measuring every stage across 72,830 calls showed the cause wasn't where I expected.

From Solana and pump.fun · 2025 – 2026

After the execution engine, I built a research pipeline around the other half of the problem: a real-time multimodal model that scores a social feed as a market signal. A post comes in, possibly with images, the model decides whether it matters, and if it does, something downstream acts on it. Speed matters a lot here, because the value of a signal decays fast.

It was lagging by around 200 to 300 milliseconds behind where I thought it should be. This post is about finding out why, and why my first guess was wrong.

The model

Some context first. At the time, the model was a fine-tuned 2B vision-language model, trained with LoRA, quantized to 5 bits, and served with llama.cpp on two AMD GPUs. (Before it there was a fine-tuned 7B, which I describe in nights and weekends on Solana.) It returns a fixed JSON shape. I had a grammar to enforce it, but the fine-tuned model produced valid JSON on its own, and the grammar cost real time per token, so it stayed off on the hot path.

That setup replaced an earlier chain of four calls to a 70B model. One call to a small, specialised model doing the whole job was faster, simpler to operate, and, once fine-tuned, good enough. The four-call chain was the first big latency win. I assumed what was left was model inference or submit time.

The serving layout matters for everything below. Text and media had separate pools: text went to multi-slot text servers on both cards, and posts with images went to four single-slot media lanes. All four media lanes ran on the first GPU.

Measuring every stage

Instead of guessing, I instrumented every stage and looked at 72,830 model calls, 210 launches and 7,400 posts of production logs.

My first suspect was the submit path, since that's where network round-trips live. It wasn't it. Metadata loading had a p50 of 10ms, the blockhash came from a warm pool on 210 of 210 launches, and transaction build had a p50 of 17ms. Submit, including confirmation, was a p50 of 282ms. Not free, but most of that is the chain, and none of it was new.

Then I split the model call itself into what the server spent working (prompt processing plus generating tokens) and everything else: the client's wall time minus the server's own time. That remainder is pure waiting, in the router, in transport, and mostly in a queue.

Text posts were fine: a p90 of 18ms waiting. Posts with media were not. Their wait was a p90 of 534ms and a p99 of 7.8 seconds, and one in five media calls waited more than 100ms before inference even started.

So the time wasn't being spent running the model. It was being spent waiting to run it.

Time spent waiting before inference, by post type (ms)Client wall time minus the server's own processing time, across 72,830 model calls (38,815 text, 34,015 media). Text posts barely waited. Posts with media waited a long time, and the tail was enormous.
  • p90
  • p99
Show data
kindp90p99
Text posts18ms59ms
Posts with media534ms7.8s

Across all calls, the time split 19.2% prompt processing, 50.5% generating tokens, and 30.3% waiting.

Queueing explains the tail

My first theory was bursts: a flurry of image-heavy posts arriving together and backing up the lanes. The data said no. Media calls started at a rate of one per second in 90.2% of the seconds that had any, and only 1 second in 6,621 had five or more starts. There were no bursts to speak of.

So I grouped media calls by how many other media calls were already in flight when each one started:

Media calls already in flight at start vs p90 wait (ms)7,391 media calls. Sample sizes: 6,244 with none in flight, 827 with one, 258 with two, 52 with three, 9 with four. The last bar is nine calls, so read it as a direction, not a number.
Show data
depthp90 wait
None82ms
One735ms
Two861ms
Three1.5s
Four2.2s

The shape is textbook single-server queueing, which was the surprise, because there were four lanes. With four independent servers, a call arriving behind one or two others should find an idle lane and barely wait. Instead, even one call ahead added most of a full service time.

The cause was the GPU, not the lanes. The driver was configured with a single hardware queue per device (GPU_MAX_HW_QUEUES=1), so all four media lanes on the first GPU submitted their work into one queue and ran one at a time at the device, however many processes existed. Four lanes on one hardware queue are one server.

A back-of-envelope check agrees. For a single server with random arrivals, the share of arrivals that find it busy equals its utilization. 84% of media calls (6,244 of 7,391) started with nothing in flight, so utilization was about 1 − 0.84 = 0.16: light load, which is why the median looked healthy. And when the server is busy, wait ≈ calls ahead × service time. A media call took about 700ms at the median, so one call ahead predicts roughly 0.7s (the measured p90 is 735ms), and four ahead predicts roughly 2.8s (the measured p90 of the nine such calls is 2.2s).

That also explains why text never suffered. Text had its own pool, and that separation is the reason its p90 wait stayed at 18ms while media's was half a second. Isolating the slow class of work is what stopped it from setting everyone's latency.

The other two costs

The queue was a tail problem. Only about 16% of media calls faced any concurrency at all, so fixing it couldn't, on its own, explain a steady few-hundred-millisecond gap. Two more things did more damage:

  • A wait for a richer copy of the post. When a post arrived without its images, the pipeline parked it for 1.5 seconds in case a fuller event followed. That hit 23.4% of text predictions (799 of 3,421), whose end-to-end median was 1,938ms against 440ms for unparked text. And when the second event came, it arrived at a p50 of 0 seconds and a p90 of 1 second, so the wait was about twice as long as it needed to be even where it helped.
  • A vision pass on the CPU, on the launch path. A separate image-classification step, used by the launch decision, ran the vision encoder on the CPU at about 420ms per post, against a 300ms budget, while the second GPU sat idle. The same pass measured 210ms on that GPU.

The fix

Three changes, in order of impact:

  • Cut the park from 1.5s to 1.0s, and let it end early: the moment the richer event arrives, it supersedes the wait.
  • Prefetch the vision pass, so it runs in parallel with the model call instead of after it. That takes its 420ms off the critical path without touching where it runs. I left it on the CPU on purpose: moving it to the second GPU would have saved latency the prefetch was already hiding, and that card hosted part of the text pool, and the machine had a history of GPU hangs.
  • Put media lanes on the second GPU. It helps because the second GPU is a second device with its own hardware queue, not because it adds lanes. More lanes on the first GPU would have bought nothing.
Before and after: count servers, not processesFour media lanes on one GPU shared one hardware queue and behaved like one server. Moving them to the second GPU gives them a hardware queue of their own. Text keeps its own pool throughout.

Twelve days and a few more small changes later, a fresh budget over 46,599 predictions looked like this:

Model call p50, before and after (ms)Before: production logs, August 5, 2026. After: 46,599 predictions over about 30 hours, August 17. The model-call stage is the like-for-like comparison; end-to-end clocks changed definition in between.
  • Aug 5
  • Aug 17
Show data
kindAug 5Aug 17
Text353ms247ms
Media725ms537ms

End to end, text posts now ran at a p50 of 278ms (p90 513ms), and the model call was 79% of the mean. Everything else was small: queueing and gating about 1ms each, the launch decision 12ms. What was left was decoding, and that was at the hardware floor: about 31 output tokens at 6.1ms each is 191ms, against a measured decode median of 203ms.

I'm deliberately quoting the model stage rather than end-to-end for media. About a third of media's end-to-end time is the image download from the CDN and the feed's own timing, both of which swing with the hour, so an end-to-end number would credit or blame me for the CDN.

Smaller things I learned along the way

One feature extractor. Training and serving share the exact same feature extraction code. Two copies that drift slightly apart is a quiet way to make a model look great offline and worse live, and I didn't want to find that out in production.

Libraries lie sometimes. The library I used to check whether the GPU supported bf16 said yes on one AMD GPU generation where it really didn't behave. Now key settings are chosen from the GPU architecture directly, not from a capability check I can't fully trust.

Most signals didn't survive. I replayed 201,796 posts against what actually happened next. Three of the four signals that looked promising didn't hold up, so I didn't change any thresholds based on them. The reasons were all about the labels, not the model: the outcome label was right-censored (recent winners hadn't had time to become winners yet), the "winner" list only contained tokens whose creators had paid to promote them, launches I had made myself were 17.5x more likely to carry an outcome at all, and every headline rested on 4 to 9 winners. The one cleanly controlled test, on 553 of my own launches with realized profit, compared the higher- and lower-scored halves within each day so market conditions were shared: the higher half won on 7 of 12 days, a sign-test p of 0.77. Within a day, the score didn't rank profit. That's a slightly deflating result, and also exactly why you replay before you tune. I wrote more about that instinct in running my own book.

What I took from it

Measure the stage, not the system. "It's slow" was true and useless. "Media calls wait in a queue, and the wait depends on how many are in flight" pointed straight at the fix.

Look at the tail, not the average. The medians looked fine the whole time. The p99 was 7.8 seconds.

Count servers, not processes. Four lanes looked like four servers in every config file I had, and behaved like one at the device. The queueing math only works if you count what actually runs in parallel.

And trust queueing theory over intuition. My intuition said the model was slow. The model was running close to its hardware floor. What was slow was everything waiting: for a device four lanes shared, and for a richer copy of a post that usually arrived within a second.

Concurrency at arrival is now one of the first numbers I record in any pipeline. It's one number per request, and it explains the tail better than almost anything else.