“AI inference engineer” is one of the highest-comp specialties in 2026 — the engineers who make GPUs go fast in production. The interview probes deep systems engineering: GPU memory hierarchy, kernel optimization, scheduling, and the particular optimizations that LLM serving demands. This guide covers the canonical topics.
Who hires for this role
- AI labs: OpenAI, Anthropic, Google DeepMind, Meta — large dedicated inference teams
- Inference platforms: Together AI, Fireworks AI, Anyscale, Modal
- Hyperscaler AI services: AWS Bedrock, Azure OpenAI, GCP Vertex
- AI-shipping product companies: Cursor, GitHub Copilot, Notion, Linear
- Open-source: vLLM project, llama.cpp ecosystem
Core knowledge: transformer inference
- Forward pass mechanics: attention, FFN, layer norm
- KV cache: what it is, why it matters, memory cost
- Prefill vs decode phase: very different compute profiles
- Output token sampling: temperature, top-k, top-p, beam search
The big optimizations to know
Continuous batching
Dynamic batching where new requests join the batch mid-flight as others finish. Massive throughput gain over static batching. vLLM popularized this. Be ready to explain why naive padding-based batching wastes GPU.
Paged attention
The KV cache is fragmented across requests. Paged attention treats memory like virtual memory pages, dramatically reducing waste. The vLLM paper (Kwon et al., 2023) is the canonical reference.
Speculative decoding
A small “draft” model proposes N tokens; the big model verifies them in one forward pass. Sub-linear speedup. Variants: Medusa (multiple heads), EAGLE, lookahead decoding. Know the tradeoffs (memory for draft model, acceptance rate).
Quantization
- FP16/BF16 baseline (already lower than FP32)
- FP8 (Hopper and beyond) — significant throughput gain
- INT8 / W8A8 — common
- INT4 / W4A16 (GPTQ, AWQ) — popular for memory-bound serving
- Tradeoff: quality vs throughput; benchmarks needed
Tensor / pipeline parallelism
- Tensor parallel: split a single layer across GPUs (NCCL-heavy)
- Pipeline parallel: split layers across GPUs (latency cost on a single request)
- Expert parallel for MoE models
- FSDP / ZeRO for training; less common in pure inference
Common interview questions
- “Walk me through how a transformer forward pass actually executes on a GPU.”
- “Why is decode memory-bound and prefill compute-bound?”
- “How would you serve a 70B model on 4 A100 80GB?”
- “Explain paged attention and why it works.”
- “How would you estimate the maximum throughput of an inference server given X GPUs?”
- “How do you choose between TP=2 and TP=4 for a given model?”
- “Walk me through speculative decoding and its limits.”
Coding rounds
Often Python or C++/CUDA-flavored:
- Implement a small attention kernel
- Optimize a piece of CPU code that does softmax
- Implement a request scheduler with priority and preemption
- Implement a memory allocator for variable-size KV cache
For senior+ roles, expect to read or write Triton or CUDA snippets.
System design rounds
Common prompts:
- “Design an inference platform that serves 100K req/s of an 8B model”
- “Design a multi-tenant inference service with priority queues and rate limits”
- “Design a serving system for a model with very long context (200K tokens)”
- “Design a streaming server with backpressure for slow consumers”
What interviewers want to hear:
- Latency budget broken down into components (queue, prefill, decode, network)
- Throughput model (tokens/sec per GPU; concurrency)
- Memory model (KV cache per token, max concurrent requests)
- Cache strategy (prompt cache, prefix cache, response cache)
- Observability (per-request latency breakdown, GPU utilization)
- Failure modes (OOM, slow GPU, NCCL stall)
Frameworks to know
- vLLM — open-source, dominant for OSS LLM serving
- SGLang — newer, optimized for structured/multi-turn
- TensorRT-LLM — NVIDIA proprietary, fastest on NVIDIA hardware
- TGI (Hugging Face) — older, less peak performance, easy DX
- llama.cpp — CPU and consumer GPU inference
- MLX (Apple Silicon)
Compensation
Inference engineers at AI labs are paid in the staff/principal IC bands — total comp $400K–$900K+ at top companies. At inference-platform startups, mid-six-figures plus significant equity. ML-systems specialty often has the highest pay band in the engineering org.
How to break in
- Read the vLLM source. Contribute meaningful PRs.
- Implement a simple LLM serving stack from scratch (decode loop, batching) for understanding
- Read the seminal papers: Flash Attention, vLLM/PagedAttention, Speculative Decoding, Medusa
- Benchmark different serving stacks on the same model and write up your findings publicly
Frequently Asked Questions
Do I need a CUDA background?
For senior+ roles, yes. For junior/mid roles, strong Python plus the ability to read CUDA goes far. The bar rises sharply at staff level.
Is this role going away as inference gets commoditized?
Unlikely. Models grow; new architectures (MoE, long-context) introduce new optimizations; inference cost is a major budget line at every AI-shipping company. The role looks durable for years.
What about training infra vs inference?
Training infra is its own specialty (FSDP, ZeRO, MoE routing, gradient checkpointing). Some engineers do both; most specialize. Inference roles are more numerous because every product team needs one.