ML Engineer Interview in the Post-LLM Era: What Changed in 2026

The ML engineer interview in 2018 looked like a slightly modified software engineering interview: standard coding rounds plus one ML-specific round on classifier design or feature engineering. The ML engineer interview in 2026 is structurally different. The default tools, default models, and default product surface area for ML engineers have shifted toward LLM-based systems, and the interview loop has evolved to match.

This piece covers what changed, what new question categories emerged, and how to prepare for the post-LLM ML engineer interview.

What stayed the same

• Standard algorithmic coding rounds (LeetCode mediums, sometimes harder).
• System design rounds (still core, with new subtopics).
• Behavioral rounds.
• The general structure of a 4-6 round onsite loop.

None of these have disappeared. The ML engineer in 2026 still needs to code, still needs to design systems, still needs to articulate behavioral stories.

What faded

• Classical ML algorithm derivations. Pre-2024 ML interviews often probed candidates on logistic regression, SVMs, random forests, gradient boosting. These topics still come up but are less central. The reason: most production ML in 2026 is LLM-based or LLM-adjacent, not classical model selection.
• Feature engineering as a core skill. Still important for tabular ML, but the typical ML engineer in 2026 spends much more time on retrieval pipelines and prompt design than on feature engineering.
• “Build a recommendation system” as the canonical ML system design. Still asked but less central. Replaced by LLM-application architecture problems.

What’s new

LLM application architecture as the default ML system design

The most common ML system design problem in 2026 is some variant of “design an LLM-based system for X.” Examples:

• Design a customer-support assistant powered by an LLM with access to the company’s knowledge base.
• Design a code-review tool that uses an LLM to suggest improvements.
• Design an enterprise search system using LLM + retrieval.
• Design an AI agent that can take actions across multiple tools.

The candidate is graded on the standard ML system design rubric (requirements, capacity, data flow, evaluation) plus LLM-specific architectural choices: which model, hosting decision, prompt design, retrieval pipeline, evaluation strategy, hallucination mitigation, cost considerations.

Retrieval-augmented generation (RAG) depth

RAG architecture has become its own subdomain. Common interview topics:

• Chunking strategies — fixed-size vs semantic vs hierarchical.
• Embedding model selection — which model, what dimensions, what cost-quality trade-off.
• Vector store choice — Pinecone vs pgvector vs Vespa vs Weaviate vs Qdrant.
• Reranking — when to add a reranker, which to use, cost implications.
• Citation grounding — how to ensure outputs reference real source material.
• Eval — how to measure RAG quality without an army of human reviewers.

Evaluation infrastructure

Pre-LLM era ML evaluation was simple: hold-out test set, AUC or F1, done. LLM evaluation is much harder, and ML engineer interviews probe this depth:

• Benchmarks vs custom evals — when each matters.
• LLM-as-judge — when it works, when it does not.
• Pairwise comparison setups.
• Adversarial / red-teaming evals.
• Distribution-shift detection in production.
• Building feedback loops from user signals.

Prompt engineering as a system-design concern

Prompt design has become a system-architecture decision in production. Interview questions probe:

• Where prompts live in the system architecture (in code, in a separate prompt store, version-controlled how).
• How to A/B test prompts safely in production.
• How to roll out new model versions without breaking existing prompts.
• When to use few-shot vs zero-shot vs fine-tuning.

Cost and latency considerations

LLM systems have non-trivial unit economics. Interviews probe:

• Token-cost-per-request math at scale.
• Latency budgets — when streaming helps, when caching helps, when smaller models help.
• Routing — using cheaper models for easy cases, expensive models for hard cases.
• Prefix caching for shared system prompts.

Fine-tuning vs prompting decisions

The classical ML training mindset is “always train your own model.” The 2026 ML engineer reflex is “use a pretrained model with prompting unless you need fine-tuning specifically.” Interview questions probe when each is appropriate:

• When prompting is sufficient (most cases).
• When fine-tuning is justified (very specific domain, very high volume, very specific format).
• When LoRA / parameter-efficient methods are right.
• When full fine-tuning is right (rarely).

The ML coding round

The coding round has shifted toward:

• Implementing pieces of an LLM application — chunking, retrieval, reranking, output parsing.
• Writing eval harnesses.
• Less from-scratch model implementation; more system-level integration.
• Some labs still ask candidates to implement attention from scratch — common but not universal.

What hiring managers grade for

The shift in 2026: ML engineer roles increasingly want generalists who can navigate the entire LLM application stack rather than specialists in one classical ML technique. The hiring manager is grading whether the candidate:

• Has shipped LLM-based features in production (or can articulate the considerations as if they have).
• Understands cost and latency constraints, not just accuracy.
• Can navigate vendor decisions (which API, which embedding model, which vector store).
• Has thought about evaluation and feedback loops.
• Can debug production issues that classical ML engineers had not seen before — hallucinations, regression after model updates, prompt drift.

How to prepare

• Build at least one substantial LLM application yourself — RAG, agent, fine-tuned model. The hiring manager wants to know you have done it, not just read about it.
• Read the major model providers’ documentation and engineering blogs (OpenAI, Anthropic, Cohere, Hugging Face).
• Drill the system design patterns: RAG, agents, model routing, evaluation harnesses.
• Stay current on benchmarks and which ones the field considers meaningful (MMLU is widely viewed as saturated; HumanEval is similar; SWE-bench Verified, MMLU-Pro, GPQA are the more current references).
• Understand the major fine-tuning methods (full FT, LoRA, QLoRA, instruction tuning, RLHF/DPO at the conceptual level).

What to skip

• Deep mathematical derivations of classical algorithms (logistic regression Hessian, SVM kernels) — rarely asked.
• Heavy feature engineering theory — less central than five years ago.
• Specific deep learning architecture trivia (number of CNN layers in AlexNet, etc.) — not asked.

Frequently Asked Questions

Do I still need to know classical ML?

Foundationally yes. You should be conversant with logistic regression, decision trees, gradient boosting, and basic neural network concepts. Deep mathematical derivations no longer come up much.

Should I know how to implement attention from scratch?

For research-track ML engineer roles yes. For applied ML engineer roles, knowing the conceptual structure (Q/K/V, multi-head, scaled dot product) is usually sufficient.

How important is fine-tuning experience?

Helpful but not mandatory for most applied ML engineer roles. RAG and prompting cover most production use cases in 2026. Specific roles working on fine-tuned models will require more depth.

What’s the most common ML engineer interview question in 2026?

Some variant of “design an LLM-based system for [domain].” This is the modern equivalent of the classical “build a recommendation system” question.

Are ML engineer salaries different from software engineer salaries?

Slightly higher on average at AI labs (10-20% premium for senior+ ML engineer roles). At FAANG, the gap is smaller but ML engineer roles still command a modest premium for senior+.