A robotics team once opened a screen by asking how I’d fit a 7-billion-parameter model onto a device with 6 GB of RAM and a hard 150 ms response budget, with no network to fall back on. There was no follow-up about training loss or attention math. The whole hour was about what you cut, what you keep, and how you prove the thing still works after you’ve mangled it. That gap is what an on-device role screens for.
Teams hiring for edge work want deployment judgment, not modeling novelty. The people running these loops have shipped models into cars, drones, hearing aids, factory cameras, and phones, and they’ve been burned by an INT8 conversion that dropped three points of accuracy the night before a demo. They want to know you’ve been burned too, or at least that you know where the fire is.
Quantization is the first question, and it’s usually a trap
Almost every loop opens with quantization because it’s the biggest single win you have and the easiest to get subtly wrong. The reasonable-sounding answer, “I’d quantize to INT8,” is where the real questions start. Post-training quantization or quantization-aware training? PTQ is a few minutes on a calibration set; QAT means retraining and often buys back one to three points of accuracy the PTQ pass lost. If a candidate reaches for QAT first, I ask what the retraining costs, because most teams try PTQ, measure, and only pay for QAT when the numbers force it.
The follow-up that separates people is calibration. INT8 works by mapping a floating-point range onto 256 integer buckets, and you pick that range by running the FP32 model over a small representative sample and watching the activation distributions. Feed it unrepresentative data and your scale factors are wrong everywhere. Then someone asks about outliers, because transformer activations carry a few enormous values that stretch the range and crush the precision of everything else. That’s the reason naive INT8 on an LLM can fall apart while it’s fine on a vision model. Strong answers reach for per-channel scales, or keeping the outlier channels in higher precision while the rest goes to INT8.
The memory math is worth carrying in your head, because interviewers will make you do it out loud.
| Numeric format | Bytes per weight | 7B model size (weights) | Typical accuracy cost | Where it ships |
|---|---|---|---|---|
| FP32 | 4 | ~28 GB | baseline | training only, never on device |
| FP16 / BF16 | 2 | ~14 GB | negligible | server and workstation GPUs |
| INT8 | 1 | ~7 GB | under 1% with good calibration | edge GPUs and NPUs with INT8 kernels |
| INT4 (GPTQ, AWQ, GGUF) | 0.5 | ~3.5 GB | roughly 1-3%, task dependent | phones and boards with 4-6 GB RAM |
Pruning and distillation, and when each is the wrong tool
Pruning comes up next, and the fault line is structured versus unstructured. Unstructured pruning zeroes individual weights and can reach 80% sparsity with under two points of accuracy loss, but it only pays off if your runtime and silicon actually skip the zeros. Most mobile hardware doesn’t, so the model is smaller on disk and exactly as slow at runtime. Structured pruning removes whole attention heads, channels, or layers, which gives you a genuinely smaller and faster dense model any runtime can use. When a candidate says “I’d prune to 90% sparsity” without asking whether the target supports sparse kernels, that’s the tell.
Distillation is the technique people underrate. Instead of shrinking a big model, you train a small student to imitate a large teacher’s outputs, and a well-distilled 1B model can beat a 3B base model on the narrow task you actually care about. On-device work is almost always narrow: wake-word detection, one defect classifier, a single assistant flow. That narrowness is what makes distillation win, and it’s why the best answers ask “what’s the actual task?” before proposing any technique at all. A candidate who wants to distill a general chatbot for a phone is solving the wrong problem; a candidate who distills a 270M model for one job on a hearing aid is thinking like the team.
The deployment stack, where good intentions meet the compiler
Compression gets you a smaller model. It doesn’t get you a fast one. The second half of these interviews is about the runtime. Export usually goes through ONNX as a framework-neutral format, then into whatever the target wants: TensorRT for NVIDIA Jetson boards, Core ML and the Apple Neural Engine on iPhones, TFLite or LiteRT and the Qualcomm Hexagon NPU on Android, ExecuTorch if you live in PyTorch. Every one of these will fall back to the CPU for any operator it doesn’t support, and that fallback is where your latency budget goes to die.
So the operator-coverage question is a real one: what happens when your model uses a layer the NPU can’t run? The graph gets split, part runs on the accelerator, part on the CPU, and the round trips between them can cost more than the compute you saved. Senior candidates talk about profiling the actual graph on the actual device, rewriting or swapping the unsupported op, and treating “does this map cleanly to the accelerator” as a model-architecture constraint they design around from the start. On a Jetson, INT8 through TensorRT is often several times faster than the same model in generic CUDA, but only when every op has a TensorRT kernel and the calibration held up.
Budgets you can’t autoscale your way out of
Cloud serving hides sloppiness behind autoscaling. On a device there is no next node. If the model plus the KV cache plus the operating system plus the rest of the app doesn’t fit in RAM, the process gets killed, and on a phone that ceiling might be 2 to 4 GB for everything at once. For an LLM the KV cache grows with context length and can rival the weights, so “supports 32k context” on a laptop can mean out-of-memory on a handset. Interviewers probe whether you treat the memory ceiling as a hard wall rather than a cost line you can pad.
Battery and heat are the constraints cloud engineers never think about. Run the NPU flat out and the device throttles, so a measured 40 ms inference becomes 90 ms after two minutes of sustained load. Any latency answer that ignores sustained versus burst performance is incomplete, and a good interviewer will sit quietly and wait for you to raise it yourself.
The air-gapped and defense angle
A growing share of these roles sit at defense, robotics, and industrial firms where the device never touches a network, by policy or by physics. That reshapes the interview. You can’t push a model update over the air, so versioning, rollback, and on-device validation carry weight they never would in a web service you redeploy hourly. You can’t phone home for telemetry, so you have to reason about failure modes you will never directly observe. And “ML deployment, not ML research” becomes literal: these teams want someone who can take a trained checkpoint and make it survive in a box in the field, not someone shopping for a new architecture.
A few questions phrased close to how they actually land in these loops:
- “You quantized to INT8 and lost four points of accuracy. Walk me through how you find out why.”
- “This model runs at 30 ms on the dev board and 200 ms on the shipping hardware. Where did the time go?”
- “You have 4 GB of RAM and need a 7B model with 8k context. Does it fit? Show your work.”
- “The NPU doesn’t support one of your layers. What are your options, ranked?”
- “How would you validate a model update on a device that can never reach the internet?”
The candidates who do well treat the model as the easy part. They’ve internalized that a checkpoint is a starting point and the hardware is the real problem, and they can hold a memory budget and a latency budget in their head while they decide which point of accuracy they’ll trade for which millisecond. If you’ve only ever deployed to a GPU behind an API, spend a weekend getting something small running on a phone or a Raspberry Pi. The moment you watch a model that “worked” fall over on real hardware is the moment these questions stop being abstract.
