System Design

Designing an LLM serving system that hits its latency budget

/

6 min read Prefill and decode, the KV cache budget, continuous batching, and autoscaling GPUs, walked through the way the interview actually goes. Read article