A 70B-parameter model in BF16 is about 140 GB of weights, and it will not fit on a single 80 GB A100. Interviewers at inference shops like Together, Fireworks, and Baseten open with exactly that: what do you do before reaching for a second GPU? The answer they want is quantization, and the follow-ups are where the interview actually happens.
Quantization stores weights, and sometimes activations, and sometimes the KV cache, in fewer bits than the 16 the model trained in. Drop the weights to 4 bits and 140 GB becomes roughly 35 GB, which fits on one card with room left for the cache. Everything after that is about what you pay for the savings and how to pay as little as possible.
Why weight-only 4-bit is the default
Token generation is memory-bound, not compute-bound. During decode the model emits one token at a time, and for each token the GPU reads every weight out of HBM. The matrix multiplies are small; the cost is moving gigabytes of weights across memory bandwidth. Shrink the weights to a quarter of their size and you move a quarter of the bytes, so decode speeds up almost in proportion before you have changed any math.
That is the logic behind W4A16: 4-bit weights, 16-bit activations. The weights get cheap to read while activations stay in FP16, which sidesteps the accuracy loss that comes from squeezing them. If an interviewer asks why not push activations to 4 bits too, the answer is outliers. A few activation channels carry values far larger than the rest, and clamping those into a 4-bit range wrecks quality. Weights behave much better, so weight-only quantization buys most of the memory win at a small cost. GPTQ and AWQ both target this scheme.
How GPTQ and AWQ actually differ
Both produce 4-bit weights. The difference is how they decide what to round and how hard to round it.
GPTQ works one layer at a time and uses approximate second-order information, a Hessian estimate over a small calibration set, to compensate for the error each rounding step introduces. When it rounds one weight, it nudges the remaining weights in the layer to cancel the damage. It is careful and it holds up well, though the calibration pass on a large model is slow.
AWQ starts from a different idea: weights are not equally important. The ones connected to high-magnitude activation channels carry most of the output quality, and there are only about 1% of them. AWQ locates those salient channels with a calibration set, then scales them so they survive quantization while the rest compress hard, folding the protection into a per-channel scale instead of a mixed-precision layout. By 2026 AWQ is the common default for production GPU serving, both because its kernels are fast and because it tends to hold reasoning quality a little better at 4 bits.
Expect a follow-up on calibration data. It should be a few hundred sequences that look like your real traffic. Calibrate a code model on prose and you measure the wrong salient channels, and the model degrades on exactly the inputs you care about. This is where people who have run the tools pull ahead of people who have only read about them.
SmoothQuant and W8A8, when compute is the wall
Weight-only quantization does nothing for prefill, the stage where the model reads the whole prompt at once and the work turns compute-bound. Long prompts, batched serving, RAG with large contexts: prefill dominates there, and speeding it up means running the matrix multiplies themselves in lower precision, which means quantizing activations.
That runs back into the outlier problem. SmoothQuant handles it by shifting difficulty from activations to weights before quantizing. It divides the activations by a per-channel factor and multiplies the matching weights by the same factor, which leaves the math unchanged but flattens the activation spikes into a range INT8 can hold. The output is W8A8, INT8 weights and INT8 activations, running on INT8 tensor cores and speeding up both prefill and decode.
Try FP8 first if the hardware has it
On Hopper and Ada cards (H100, L40S, and newer) there is an easier route that often beats the integer methods: FP8. An 8-bit float keeps an exponent, so it absorbs the wide activation ranges that break INT8 without any smoothing transform. The standard advice on those GPUs is to try FP8 first for the best accuracy-to-throughput ratio, drop to INT8 SmoothQuant next, and go to AWQ or GPTQ at 4 bits only when you need more memory savings than FP8’s 2x.
A number worth quoting: on Qwen3-32B, BF16 to INT8 measured around a 0.04% drop on standard benchmarks, and even 4-bit kept roughly 98% of baseline reasoning on MMLU-Pro. Quantization is rarely the quality disaster people brace for. The disasters come from quantizing the wrong thing, or from shipping without an eval.
The KV cache is the third thing to quantize
Weights are a fixed cost. The KV cache grows with every token and every concurrent request, and on long-context workloads it can take more memory than the weights. Quantizing it is often what lets you raise batch size or context length without a second GPU.
INT8 KV cache is close to lossless in practice. INT4 KV carries a small, measurable loss and earns its place when memory is the hard limit. On Hopper and Ada, FP8 KV cache tends to beat INT8 for the same reason FP8 weights do: the exponent covers the range. Interviewers like this probe because it separates people who picture quantization as one knob from people who know it applies to three separate things, weights, activations, and the cache, each with its own tradeoff.
GGUF and the laptop question
When the role touches on-device or local inference, GGUF shows up. It is the file format llama.cpp uses, and its K-quant variants (Q4_K_M and its siblings) mix bit depths across the model: sensitive layers such as attention get 5 or 6 bits, feedforward blocks get 4. That mixed allocation is why GGUF reaches better quality per bit than flat 4-bit, and why it is the format people pick for running a model on a MacBook or a CPU box. The catch is that its kernels are tuned for CPU and Apple Silicon, so on a datacenter GPU you would reach for AWQ or FP8 instead.
The comparison you should be able to draw from memory
| Method | Precision scheme | Needs calibration data | Best fit | Typical quality hit at target bits |
|---|---|---|---|---|
| GPTQ | W4A16, 4-bit weights, Hessian error compensation | Yes | Memory-bound GPU decode | Small at 4-bit |
| AWQ | W4A16, 4-bit weights, salient channels scaled | Yes | Default GPU serving in 2026 | Small, edges out GPTQ at 4-bit |
| SmoothQuant | W8A8, INT8 weights and activations | Yes | Compute-bound prefill on INT8 cores | Near-lossless at 8-bit |
| FP8 | 8-bit float weights and/or activations | Minimal | Hopper and Ada GPUs, first thing to try | About 0.04% on INT8-class tests |
| GGUF K-quants | Mixed 4 to 6-bit weights per layer | No | CPU, Apple Silicon, local runs | Strong quality per bit |
| KV cache quant | INT8, INT4, or FP8 cache | No | Long context and high batch size | INT8 near-lossless, INT4 slight loss |
Questions to expect
The phrasings vary by company, but the shape is consistent:
- You need to serve Llama-3-70B on one H100. Walk through your quantization plan and how you would confirm it worked.
- Why does 4-bit weight-only quantization speed up decode but not prefill?
- Your INT8 model passes a perplexity check but fails a downstream coding task. What went wrong?
- When would you ship GGUF instead of AWQ?
The strongest answers treat quantization as a measurement problem. Anyone can run the quantizer. The person who gets the offer picks the scheme from the bottleneck (memory versus compute), matches the calibration data to real traffic, and then proves the model still answers correctly on the tasks that pay the bills. Say you would keep the FP16 model around to A/B against for a week, and you sound like someone who has shipped one.
