# Designing an LLM serving system that hits its latency budget

Source: https://www.techinterview.org/post/3233476388/designing-an-llm-serving-system/
Updated: 2026-07-15 · techinterview.org

An interviewer slides over the prompt: design the inference service behind a chat product, a few thousand requests a second, keep the responses fast and the GPU bill sane. The candidates who do well spot something in the first couple of minutes that the rest miss. You are not designing one system. Generating a reply happens in two phases that stress completely different parts of the hardware, and most of the decisions that follow fall out of that split.

## Prefill and decode are two different machines

When a request arrives, the model first reads the entire prompt in a single forward pass. This is prefill. Every prompt token is processed in parallel, the matmuls are large, and the GPU compute units run hot. Prefill produces the first output token and fills the key/value cache for the whole prompt. Time to first token, the thing a user feels as "did it hear me," is dominated by how long prefill takes, which scales with prompt length.

Then the model generates the reply one token at a time. This is decode. Each step feeds the previous token back in, reads the full model weights and the growing KV cache out of high-bandwidth memory, and emits exactly one token. The math per step is tiny, so the compute units sit mostly idle while the memory bus does all the work. Decode is memory-bandwidth-bound. Inter-token latency, the pause between words as they stream out, lives here.

That asymmetry is the whole game. Prefill wants raw compute. Decode wants memory bandwidth and a lot of concurrent sequences to keep the bus busy. Put them on the same GPU and a long prefill for one incoming request stalls the token stream for every user already mid-decode. Say that out loud and the interviewer knows you have actually run one of these.

## The KV cache is the budget everything spends against

The KV cache is why LLM serving is a memory problem before it is anything else. For every token in context, every layer stores a key and a value vector so the model does not recompute attention over the whole history at each step. That memory grows with context length and with the number of concurrent requests, and it competes with the model weights for the same GPU.

The per-token size is easy to estimate, and interviewers like watching you do it: 2 (one key, one value) times layers times KV heads times head dimension times bytes per number. Grouped-query attention is the reason the numbers below are survivable at all. It shares KV heads across query heads and cuts this cost by roughly eight times versus classic multi-head attention, which is exactly why every recent model ships with it.

| Model (GQA, BF16 KV cache) | Layers | KV heads × head dim | KV cache per token | Per request at 128K context |
| --- | --- | --- | --- | --- |
| Llama 3.1 8B | 32 | 8 × 128 | ~0.13 MB | ~16 GB |
| Llama 3.1 70B | 80 | 8 × 128 | ~0.31 MB | ~42 GB |
| Llama 3.1 405B | 126 | 8 × 128 | ~0.49 MB | ~66 GB |

Read the last column and the design pressure is obvious. One Llama 3.1 70B request at full context wants about 42 GB of KV cache, while the weights alone are 140 GB spread across at least two 80 GB cards. The number of requests you can batch is capped by whatever memory is left over, not by compute. Quantizing the KV cache to FP8 roughly halves that column and buys you twice the batch.

## Why static batching leaves the GPU idle

The naive way to batch is to collect N requests, run them together until they all finish, then take the next N. It falls apart because replies have wildly different lengths. A batch with one 800-token answer and fifteen 40-token answers runs at the pace of the slow one while fifteen slots sit finished and wasted. On a decode workload that already struggles to keep the GPU busy, that is most of your throughput gone.

Continuous batching, which vLLM popularized off the Orca paper, works at the granularity of a single decode step. After every step it drops finished sequences and pulls waiting ones into the batch, so the GPU runs a full batch on every iteration. On the same H100 this is the difference between a naive PyTorch loop and something serving three to five times the traffic. The catch, again, is that how many sequences you can hold is set by KV cache memory.

PagedAttention is what makes that memory usable. Instead of reserving a contiguous block for each request maximum possible context, it stores the KV cache in fixed-size pages scattered around memory, the way an operating system pages RAM. You allocate a page only when a request actually needs it, fragmentation nearly disappears, and requests that share a prefix (the same system prompt across thousands of chats) can point at the same cached pages instead of recomputing them.

## Pick the SLOs before you draw any boxes

A serving design has no correct answer until you name two latency targets, and reaching for them early is one of the clearer signals of experience. Time to first token is what makes a chat feel responsive; a few hundred milliseconds reads as instant, a couple of seconds reads as broken. Inter-token latency needs to stay faster than the user reads, somewhere around 50 to 100 milliseconds per token or better, so the text never visibly stutters.

Those two numbers fight throughput. A bigger batch drives down cost per token and lifts GPU utilization, and it also lengthens every prefill and every decode step, which pushes both latencies up. You are choosing an operating point on that curve, not a right answer. Expect the follow-up: "your p99 time to first token doubles under load, what is happening." The usual culprit is a prefill queue backing up behind long prompts, or short interactive requests stuck waiting behind a giant batched prefill.

## Autoscaling when a cold start is 90 seconds

This is where people who have only served stateless web traffic get caught. You cannot autoscale a GPU fleet the way you scale a web tier. A 70B model is 140 GB of weights to pull onto the GPUs, plus CUDA graph capture and warm-up, and a cold replica can take tens of seconds to a couple of minutes before it serves its first token. By the time a reactive scale-up is ready, the traffic spike is over.

So you scale on the signals that lead demand rather than trail it: queue depth, pending tokens waiting for a slot, batch occupancy. CPU utilization, the default metric everyone reaches for, means nothing here because the bottleneck is memory bandwidth. Keep a warm pool of spare replicas sized to your expected burst, over-provision headroom on purpose, and treat scale-to-zero as something you do for tolerant batch jobs, never for an interactive endpoint. A smaller, cheaper model tier can soak up overflow while the big replicas come online.

## Where strong candidates take it next

Once the base design holds together, the follow-ups separate people who read papers from people who have shipped. A few that come up constantly:

- "Prefill is stalling your decode latency. How do you fix it without more GPUs?" Chunked prefill breaks a long prompt into slices interleaved with decode steps; at real scale you disaggregate, running prefill and decode on separate GPU pools (DistServe and Splitwise are the reference points) and shipping the KV cache between them.

- "Decode is memory-bound and the GPU is idle. Can you use that spare compute?" Speculative decoding: a small draft model proposes several tokens, the big model verifies them in one pass, and you get two to three times faster decode whenever the draft guesses right.

- "The model does not fit on one GPU. Now what?" Tensor parallelism splits each layer across GPUs with an all-reduce per layer, which only pays off with fast interconnect like NVLink; across nodes you fall back to pipeline parallelism and eat the bubble.

The candidates who land it are not the ones who name the most techniques. They are the ones who tie every knob back to the same two facts: generation is a compute-bound prefill bolted to a memory-bound decode, and the KV cache is the memory budget that everything else is spending. Draw those first, put a latency target on the board, and the rest of the design has somewhere to stand.