Written October 2026, looking back at Systems22 min read
Sixteen months at IBM
CI/CD for Watson, then telemetry, then building Watson Code Review, an LLM review service on RabbitMQ and OpenShift. KV caching, when to re-run a review, how results reach developers, and how it scaled.
From IBM · May 2024 – Aug 2025
My IBM internship ran from May 2024 to August 2025, remote out of Markham, and it went in three stages: CI/CD for Watson, then telemetry, then building Watson Code Review, a service that reviews pull requests with an LLM. Each stage handed me a bigger piece of the system than the last.
- Stage 1CI/CD for WatsonBuild, test, and release pipelines for Watson services on OpenShift, plus an SDK and CLI for spinning up test environments.
- Stage 2TelemetryA Go client for watsonx, OpenTelemetry across services, error-biased sampling, and performance work on the agent that collects it all.
- Stage 3Watson Code ReviewAn LLM reviewing pull requests, from a side project to a service other teams relied on.
Stage 1: CI/CD for Watson
I started on the pipelines that build, test, and ship Watson services. A lot of engineers depend on those pipelines working, and almost nobody thinks about them until they don't.
The work was about making releases boring:
- One pipeline system. I moved repositories from Travis CI onto Jenkins, carrying build history and artifacts across, without a window where teams couldn't ship.
- Disposable test clusters. Integration tests ran on lightweight Kubernetes (K3s) clusters created per run, with the dependency images they needed managed by the pipeline instead of by hand.
- Faster feedback. Parallel stages and caching so CI stopped being the slowest part of everyone's day.
- Environments on demand. Engineers needed test VMs and OpenShift clusters from IBM's internal provisioning cloud, mostly through copied scripts. I owned an inner-source Go SDK for that API and a CLI on top of it (built on Cobra, with tab completion), so an environment became one typed call or one command.
Two of those choices are worth unpacking, because each has a cost:
- Why a cluster per run instead of one shared test cluster. A shared cluster accumulates state: a namespace a failed run never cleaned up, a CRD a previous branch installed, an image one test left behind. Tests then pass or fail depending on what ran before them, which is the worst kind of flaky because rerunning "fixes" it. A cluster created per run starts from the same empty state every time, and two pipelines never contend for it. The price is paying cluster start-up on every run, which is why it was K3s (a single, small Kubernetes distribution) rather than a full OpenShift install per run, and why the dependency images each run needed were managed by the pipeline rather than by whoever set up the test.
- Build once, promote the same artifact. The image that passed integration tests is the one that gets promoted, not a rebuild of the same commit for staging. A rebuild can pull a different base image or dependency than the one you tested, so "it passed CI" stops meaning anything about what shipped. An image digest is a hash of the image's content, so promoting by digest rather than by tag makes the tested thing and the shipped thing provably the same bytes.
The lesson: infrastructure work is trust work. A pipeline that works is invisible, and that's the goal.
Stage 2: Building out telemetry
Once I knew how the services shipped, I moved to understanding how they behaved. I added OpenTelemetry tracing across Watson services so a single request could be followed through every hop it made.
The interesting decision was which traces to keep. Uniform sampling throws away the rare failures you actually need, so I pushed for error-biased tail sampling: keep every failing or slow trace, and sample healthy ones down. I wrote about that in detail in keep the failing traces.
A client for the request path
The same instinct as the provisioning SDK came back on the serving side, where I wrote a Go client for calling models on watsonx through IBM Cloud's auth. It kept warm, pooled connections instead of paying a fresh TLS handshake on every request, had sane timeouts and retry rules by default, and returned errors you could act on without parsing strings. Other teams picked it up.
The connection pool is the part that's easy to undervalue. A fresh HTTPS connection costs a TCP handshake (one round trip) plus a TLS 1.3 handshake (one more) before the first byte of the request is sent, so two round trips of pure overhead per call. As an estimate with an illustrative 30ms round trip, that's about 60ms added to every request made on a cold connection, and it lands on every caller who builds their own client the quick way. A pooled client pays it once per connection and then reuses it. The defaults matter for the same reason: a library whose safe behaviour is opt-in gets used unsafely, so the timeouts and retry rules were on unless you turned them off. What I learned is that the API is maybe half of a library. The other half is the defaults, the docs, and answering people quickly when they hit a sharp edge.
The telemetry agent
I also worked on the agent that collects and exports telemetry. Many producer threads handing records to one exporter is a classic contention point, so the handoff became a lock-free queue, record layout was made cache-friendly after profiling with perf and VTune, and collectors got failover so a crashed primary hands over to a standby instead of dropping data.
The reasoning behind each piece:
- Why lock-free. With a mutex, every producer that arrives while another holds the lock has to wait, and under heavy contention that wait can mean the thread is parked by the kernel and woken later. A multi-producer, single-consumer queue replaces the lock with one atomic compare-and-swap per enqueue, so a producer that loses a race retries immediately instead of sleeping, and the single consumer never contends with anyone on its side of the queue.
- Why layout matters. CPUs move memory in 64-byte cache lines. If two fields written by different threads share a line, every write by one thread invalidates the other's copy, and the line bounces between cores even though the threads never touch the same data (false sharing). Laying records and queue indices out so that each hot field written by a different thread sits on its own line removes that traffic. That kind of problem doesn't show up in the code; it shows up in a profile as cache misses, which is why the profiling came first.
- Why a standby. An exporter with one destination turns a collector restart into a gap in the data, at exactly the moment (something restarting) when you want the data most.
That telemetry ended up being the foundation for the next stage: I couldn't have run an LLM service responsibly without it.
Stage 3: Watson Code Review
Watson Code Review started as something I built on the side: a GitHub integration where a fine-tuned Granite code model, served on watsonx, reviews a pull request and leaves comments on it. It was later adopted as its own product inside IBM.
The model was the easy part to describe. The engineering was everything around it: getting large diffs through a fixed context window, keeping GPU time affordable, deciding when to review again, getting results back to developers in a form they'd trust, and scaling the whole thing on OpenShift.
The architecture
The service was split into stages connected by RabbitMQ queues, each with its own pool of workers (handlers):
- Ingest receives GitHub's webhook, verifies its signature, ignores deliveries it has already seen, records a review job in PostgreSQL, and returns immediately. GitHub expects a fast response, and nothing slow belongs on that path.
- Diff prep fetches the changes, splits them into chunks, and builds the prompts.
- Inference sends chunks to the model, batching requests together.
- Publish turns model output into comments and status updates on the pull request.
Splitting it this way meant each stage could scale on its own and fail on its own. A burst of large pull requests backs up the inference queue, not the webhook endpoint. A GitHub API hiccup backs up publishing, not the GPUs.
RabbitMQ fit this better than the two obvious alternatives, each of which would have had a real cost for this workload:
- Kafka is a log: consumers read partitions in order, and parallelism is capped by the partition count. That's right for event streams, but a review queue is a work queue, where messages take very different amounts of time. In a partition, one slow review holds up every message behind it, and retrying one message means building your own retry topics. RabbitMQ gives per-message acknowledgement, redelivery, and dead-lettering natively, and any free worker can take the next message.
- PostgreSQL as the queue (rows claimed with
SELECT ... FOR UPDATE SKIP LOCKED) would have saved a component, and at low volume it works. The cost is that workers poll instead of being pushed work, every claim and completion is a write to the same hot table the job records live in, and backpressure (how many messages a worker may hold) is something you build rather than configure. Postgres stayed the system of record for jobs and results; RabbitMQ moved the work.
A few rules every handler followed:
- Acknowledge after the work, not before. A message is acked only once its result is safely written, so a worker crashing mid-task means the message is redelivered, not lost.
- Idempotent by design. Work is keyed on repository, pull request, and commit SHA, so a redelivered message finds the result already there and does nothing.
- Bounded retries, then a dead-letter queue. Transient failures retry with backoff; anything that keeps failing goes to a dead-letter queue for a human, instead of looping forever.
- Limited prefetch. Each worker only holds a few unacknowledged messages, so work spreads across workers instead of piling onto whichever one connected first.
Those rules interact, and the interactions are where the design choices are:
- Prefetch is a latency trade. A worker with a high prefetch has its next messages already on hand, which hides the broker round trip and suits short tasks like publishing. For long tasks it's the opposite: messages sit in one busy worker's buffer while another worker is idle, and if that worker dies they all wait for redelivery. So prefetch belongs to each stage rather than to the system: smallest where tasks are long, larger where they're short.
- Ack-after-work has a ceiling. RabbitMQ closes a consumer's channel if it holds a delivery unacknowledged for longer than its consumer timeout (30 minutes by default), and requeues the message. A message that represents "review this entire pull request" can drift toward that limit on a big change; a message sized to a chunk of work stays far below it, and a crash wastes one chunk of GPU time rather than a whole review.
- Idempotency covers the database, not GitHub. Keying results on repository, pull request, and commit means a redelivered message finds its result and stops. The window it can't close on its own is a side effect outside the database: a publish worker that posts a comment and crashes before recording it will post again on redelivery. That's why publishing looks for its own earlier comment before writing (see "update, don't duplicate" below): the check has to be against GitHub's state, because that's where the side effect lives.
Fitting a diff into a context window
A model has a fixed context window. Send a large diff whole and the tail is silently cut off, which means part of the change never gets reviewed and nobody knows.
So diff prep splits every pull request by file and by hunk into chunks that fit a per-model token budget, keeping enough surrounding lines for the model to understand each change. A large pull request becomes several requests instead of one truncated one.
The budget for the diff itself is whatever the window has left after everything else in the prompt, and after room for the answer:
chunk budget = context window - shared prompt - reserved output
illustrative: 8,192 - 2,000 - 1,000 = 5,192 tokens for the diff and its context
Two things follow from that line. First, the output reservation is not optional: a model that runs out of room mid-answer truncates the review just as surely as a truncated input does, only at the other end. Second, every token added to the shared instructions comes out of every chunk's budget, so a longer prompt means more, smaller chunks per pull request. That cost is paid in context space, not compute, because the shared part is cached (next section). The window, not the GPU, is what limits how much instruction a review can carry.
Splitting on file and hunk boundaries, rather than at a fixed token count, is what keeps a chunk reviewable on its own: a cut in the middle of a function gives the model half a change and no way to know the other half exists. The limit of this design is the change that only makes sense across files, like a renamed function and its callers. Each chunk sees its piece, and no single request sees the whole change.
KV caching: making the shared part free
This is the part I found most interesting.
Every review prompt had a large part that was the same across requests (the instructions, the review guidelines, the output format) and a small part that was unique (the diff chunk). If the unique part came first, nothing could be reused. So the prompt layout was designed around the cache:
A few details made the difference:
- Most stable first. Global instructions, then per-repository guidelines, then file context, then the chunk. Each layer is shared by a smaller group of requests, so the cached prefix is as long as possible for each.
- Byte-for-byte identical. Anything that varies, like a timestamp or a reordered list, breaks the prefix. The shared parts were generated deterministically so the cache actually hit.
- Send related chunks to the same worker. Chunks from the same pull request share the most prefix, so routing them together means the cache on that worker is already warm.
What the cache saves, and what its hit rate depends on.
Prefill is compute-bound, and at prompt lengths of a few thousand tokens its cost is roughly proportional to the number of tokens it has to compute. A cache hit on a prefix means those tokens are skipped, so the saving is simply the share of the prompt that was already cached. As an estimate with illustrative sizes (not measured ones): 1,500 tokens of review instructions, 500 of repository guidelines, 800 of file context, and a 1,200-token chunk.
Show data
| case | Tokens prefilled |
|---|---|
| Chunk first (nothing reusable) | 4,000 |
| Layered, new repository | 2,500 |
| Layered, repository warm | 2,000 |
| Layered, same file's next chunk | 1,200 |
In the best case a request prefills 1,200 of its 4,000 tokens, 30% of the uncached cost. In general the expected work per request is the unique tail plus each shared layer weighted by how often it misses:
prefill per request = chunk + (1 - h_instr) x instr + (1 - h_repo) x repo + (1 - h_file) x file
So the cache's value is entirely in those hit rates, and three things set them:
- The prefix has to be identical. One differing token ends the match, and everything after it is recomputed. Engines that cache in fixed-size blocks (vLLM, for example, hashes blocks of 16 tokens by default) only reuse whole blocks, so the reusable part is the shared prefix rounded down to a block boundary. That's why the shared parts were generated deterministically: the cache can't tell a reordered list from new content.
- The request has to land where the cache is. Each inference replica has its own cache. A request for a repository whose prefix is warm on replica A, routed to replica B, is a miss. That's the reason for routing a pull request's chunks together.
- The entry has to survive until it's needed. Cached blocks live in the same GPU memory as every in-flight request's working state, and under load the least recently used ones are evicted. A repository's guidelines stay warm while it gets steady traffic and go cold when it doesn't, so the hit rate on the per-repository layer is a function of traffic, not of the design. The global instructions are shared by every request and effectively never go cold.
There's a memory saving too, in engines that store a shared prefix once rather than once per request. With an illustrative 8B model using grouped-query attention (32 layers, 8 key-value heads of 128 dimensions, 16-bit values), the cache costs:
per token = 2 (key and value) x 32 layers x 8 heads x 128 dims x 2 bytes = 128 KiB
2,000-token shared prefix = 2,000 x 128 KiB ≈ 250 MiB
Sixteen requests in a batch that share that prefix store it once instead of sixteen times, which frees about 3.7 GiB, room for roughly 30,000 more tokens of unique context. Memory is what caps batch size, so prefix sharing also means bigger batches.
Why batching pays.
Combined with dynamic batching, where a worker collects requests for a few milliseconds and runs them through the GPU together, this is what made the service affordable. Batching trades a small amount of latency for much better GPU throughput, and the wait is capped so no single review sits waiting for a batch to fill.
The reason batching works is that generating a token is memory-bound, not compute-bound. Every decode step has to read all of the model's weights from GPU memory, whether it's producing one token or one for each of 32 requests. As an estimate with public hardware numbers and an illustrative model: an 8B model in 16-bit weights is 16 GB, and an A100 80GB reads memory at about 2.0 TB/s, so one step can't take less than about 8ms. With one request, that's roughly 125 tokens per second, and the GPU's arithmetic units mostly wait. Add requests to the batch and the weight read is shared; what grows is each request's own cache, about 500 MiB for a 4,000-token context at the size above.
Show data
| Requests per batch | Tokens per second, all requests |
|---|---|
| 1 | 123 |
| 4 | 451 |
| 8 | 808 |
| 16 | 1,338 |
| 32 | 1,991 |
Going from 1 to 32 requests per batch multiplies throughput by about 16 in that model, while each individual request slows from about 123 to 62 tokens per second, because each step now also reads 32 requests' caches. That's the trade in one line: much cheaper per token, somewhat slower per review. The batching wait adds at most its cap to a review that already takes seconds of generation, which is why it's capped in milliseconds rather than tuned to fill batches.
When to review again
Pull requests change. Deciding when to re-run a review mattered as much for cost as for usefulness:
- Supersede, don't stack. A new push makes any in-progress review of an older commit obsolete, so it's cancelled rather than finishing and posting stale comments.
- Debounce bursts. Developers often push several commits in a row; waiting a moment avoids reviewing each one.
- Review the change, not the whole PR. Chunks are identified by their content, so a chunk that hasn't changed since the last review reuses its earlier result.
- Skip what shouldn't be reviewed. Draft pull requests, generated files, and lockfiles are skipped, and a developer can ask for a fresh review explicitly when they want one.
The cost model behind those rules is simple. If a pull request splits into n chunks and a new push changes k of them, an incremental re-review costs about k / n of a full one. As an illustration: a fix-up commit that touches 2 of a pull request's 20 chunks costs about 10% of reviewing it from scratch, and a pull request reviewed once in full and then after four such pushes costs about 1.4 full reviews (1 + 4 x 0.1) instead of 5. The less a push changes, the more the content-addressing saves, and small follow-up pushes are the common case in review.
Each rule also has a cost of its own, and a limit:
- Debouncing adds its window to every review. A debounce of
wseconds delays even a single, lone push byw, and only pays off when pushes arrive closer together thanw. That's why the window was short: long enough to absorb a burst of pushes in a row, short enough not to be what a developer is waiting on. - Cancellation only saves work it reaches in time. A stale job is recognised when its commit is no longer the pull request's head, and each stage can check that before starting work. A chunk already running on the GPU when the new push arrives finishes anyway; superseding stops the work that hasn't started and stops stale comments from being posted.
- Content-addressing assumes a chunk stands alone. A chunk that hasn't changed reuses its result even if code around it, in another file, changed in a way that matters to it. That's the same cross-file blind spot as chunking itself, and it's the case the explicit "review again" request exists for.
Getting results to developers
A review is only useful if developers trust it, and trust comes from clarity:
- A status on the pull request that moves from queued, to in progress, to complete, so nobody wonders whether it's running.
- Inline comments anchored to the exact lines they're about, plus one summary comment.
- Update, don't duplicate. On a re-run, earlier comments are updated or resolved rather than reposted, so a pull request doesn't fill up with repeats.
- Honest failures. If a review couldn't run, the status says so plainly, instead of silently showing nothing.
- Traceable. Every review carries its trace ID, so when someone asks "why did it say that?", I could follow that exact request through every stage.
Scaling it on OpenShift
Each stage ran as its own deployment on OpenShift, so it could be scaled and rolled out independently.
- Scale on backlog. CPU usage is a poor signal for a queue-driven service: a worker waiting on the model looks idle while work piles up. Queue depth is the honest signal, so it drove scaling.
- Different limits per stage. The CPU-bound stages scale wide cheaply. GPU inference scales within a fixed budget, so when the budget is reached, work queues instead of costs growing without limit.
- Lanes by size. Small pull requests shouldn't wait behind a huge one, so work was separated by size, the same idea I later used for text and media lanes in my own ML pipeline.
- Graceful shutdown. On scale-down or deploy, a worker stops taking new messages, finishes or returns what it holds, and only then exits, so rollouts don't lose work.
- Health checks that mean something. A worker reports ready only once it can reach the queue and its downstream dependencies, so traffic never goes to a pod that can't do the work.
The scaling rule, and what the GPU cap costs. Scaling on backlog is proportional: the target is a number of waiting messages per worker, and the replica count follows the backlog divided by that target, clamped between a floor and a ceiling. That's the same shape as Kubernetes' own rule, desired = ceil(current x observed / target), applied to queue depth instead of CPU. For the CPU stages the ceiling is generous. For inference, the ceiling is the GPU budget, and above it the cost of a burst moves from money to waiting time. The waiting time is just the backlog divided by the drain rate:
wait ≈ backlog / (replicas x chunks per second per replica)
illustrative: 600 chunks queued / (4 replicas x 2 chunks/s) = 75 s
At the ceiling, doubling the backlog doubles the wait. That is the deliberate trade: a burst of large pull requests makes reviews arrive later, rather than making the bill arrive bigger. Size lanes exist because of the same formula: without them, one pull request with hundreds of chunks puts all of them in front of a two-chunk change, and the small change waits for the whole backlog.
Graceful shutdown is about wasted work, not lost work. Because messages are acknowledged only after their result is written, a pod killed mid-task loses nothing: the broker requeues the message. What a hard kill wastes is the GPU time already spent on it. Kubernetes gives a pod 30 seconds by default between the stop signal and the kill, so a worker that stops consuming on the signal and finishes its in-flight chunks inside that window wastes nothing on a normal rollout.
Cache affinity and load spreading pull in opposite directions. Routing a pull request's chunks together keeps the prefix cache warm, but it also means one huge pull request lands on one replica while others sit idle. The prefix cache argues for stickiness, and the queue argues for any free worker taking the next message. Lanes by size are part of the answer, since the stickiness costs the most on exactly the pull requests big enough to have their own lane. The general rule I took from it: affinity is worth it until the load it concentrates costs more than the cache misses it saves.
What working at that scale felt like
The first few months were humbling. On my own projects I owned everything. At IBM I owned a small piece of something many people depended on, and most decisions involved people I'd never met.
- Write things down. On a remote team across time zones, a decision that isn't written down didn't happen.
- Reviews are a conversation. I started treating code review as the cheapest way to borrow someone else's experience.
- Scope is negotiated. "What's the smallest version of this that helps someone this month?" saved me more time than any tool.
- Ownership includes the boring parts. Owning a pipeline, an SDK, or a service means answering the message about why it's broken, whenever it arrives.
What's next
While I was at IBM, my evenings went somewhere else entirely: Solana. And as this internship wound down, I started at Print.World. Both made more sense because of what IBM taught me about owning something other people depend on.