AI agent infrastructure has become its own system design topic in 2026 interviews. The shift mirrors what happened with microservices in the early 2010s — once a niche topic, now a core senior+ system design question because production deployments have made it real engineering work. AI labs and the AI-native startups built on top of them ask agent-infrastructure questions routinely; some traditional tech companies are starting to.
This piece covers what AI agent infrastructure means as an interview topic, what the canonical questions look like, and how to prepare for them.
What “AI agent” means in this context
For interview purposes, an AI agent is a system that:
- Receives a goal from a user or another system.
- Calls tools (search, code execution, API calls, file reads, browser automation) iteratively to make progress on the goal.
- May persist state across multiple steps (memory, intermediate results).
- Returns a result, which may be itself a multi-step trace rather than a single answer.
Classical LLM serving (one prompt in, one response out) is a simpler problem. Agents add iteration, tool calls, persistent state, and longer traces — and each addition changes the engineering problem materially.
The canonical interview question
“Design an infrastructure platform for running AI agents at scale. Customers come with their own agent code and prompts; we host the orchestration, tool execution, and observability.”
This is the modern equivalent of “design a URL shortener.” Small enough to fit in 60 minutes, deep enough to test architectural reasoning, and obviously relevant to 2026 engineering work. Variants:
- Design a coding-agent platform like Cursor’s background agents or Claude Code’s agentic mode.
- Design infrastructure for a customer-support agent that handles 1M conversations per day.
- Design a research-assistant agent platform where individual traces can run for 30+ minutes.
- Design tool execution infrastructure that lets agents safely run arbitrary code.
The architectural elements
1. Tool use orchestration
Agents call tools repeatedly. The infrastructure question: how do you route, queue, and rate-limit those calls?
- Per-agent rate limits to prevent runaway loops.
- Per-tool rate limits to protect downstream services.
- Tool call routing — fast tools execute synchronously, slow tools queue.
- Concurrency limits within a single agent trace to prevent excessive parallelism.
- Tool failure handling — retries, backoff, fallback paths.
2. Persistent memory and state
Long-running agent traces accumulate state — intermediate results, tool outputs, reasoning chains. Infrastructure questions:
- Where does this state live? In-memory of the orchestrator? Durable storage?
- How do you compress or prune long traces to fit context windows?
- How do you resume an agent that crashed mid-trace?
- How do you handle agents that need to remember across separate user sessions (long-term memory)?
3. Sandboxing and security
Agents that execute code, browse the web, or call APIs need isolation. Infrastructure questions:
- Code execution sandboxing — Firecracker, gVisor, container-based isolation.
- Network isolation — what can the agent reach? How do you prevent exfiltration?
- Filesystem isolation — agent can write to its scratch space but not affect other agents.
- Time and resource limits — how do you stop a runaway agent that has consumed too much CPU or made too many tool calls?
4. Billing and cost accounting
Agent traces have variable cost — short traces might be a few cents, long ones tens of dollars. Infrastructure questions:
- How do you bill per-trace when the trace cost is not known until completion?
- How do you cap a customer’s spend mid-trace?
- How do you allocate cost between LLM tokens, tool calls, sandbox compute, and storage?
- How do you give customers visibility into where their cost is going?
5. Observability and debugging
Agent traces are hard to debug. Infrastructure questions:
- How do you record the full trace (every prompt, response, tool call, intermediate state) without exploding storage cost?
- How do you let developers replay a trace to investigate a bug?
- How do you build alerting for agents that are misbehaving (looping, exceeding budget, hallucinating tool calls)?
- How do you handle privacy when traces include user data?
6. Multi-tenant isolation
Customers’ agents must be strictly isolated:
- Agent code from customer A cannot affect customer B’s agents.
- Prompts, traces, and tool outputs are tenant-scoped.
- Tool credentials (customer A’s Stripe key) are accessible only to customer A’s agents.
Common candidate mistakes
- Treating it as classical LLM serving. Agents are not just “LLM with extra steps.” The persistent state, tool use, and longer traces require different architecture.
- Hand-waving sandboxing. “We use Docker” is not enough. Senior+ candidates should articulate the specific isolation guarantees and the threat model.
- Missing the cost-control angle. A platform that lets customers spin up agents without spend limits is a liability. Strong candidates raise cost control unprompted.
- Ignoring the long-trace problem. Many candidates assume traces fit in a context window. They do not. The compression / pruning / resumption story is part of the design.
- Underestimating observability. Agent debugging is genuinely hard; the infrastructure has to support it as a first-class concern, not an afterthought.
What strong candidates do
- Start with capacity estimation. How many agents per second? How long does a typical trace take? What’s the storage footprint per trace?
- Walk through the trace lifecycle from start to completion, articulating where state lives at each step.
- Treat tool execution, sandboxing, and billing as first-class subsystems rather than afterthoughts.
- Discuss failure modes explicitly — what happens when an agent loops, when a tool fails, when the orchestrator crashes mid-trace.
- Bring familiarity with real systems (Anthropic’s Claude Code, OpenAI’s Assistants API, LangChain Smith for observability) without name-dropping for its own sake.
How to prepare
- Build at least one substantial agent yourself. Use Anthropic’s tool use, OpenAI’s function calling, or a framework like LangGraph. Get hands-on with the lifecycle.
- Read the engineering blogs of platforms that operate agents at scale — Anthropic, OpenAI, Replit, Cursor, LangChain.
- Study one production agent platform’s public architecture details. The best 2026 references are publicly-discussed pieces of Anthropic’s Claude Code infrastructure and OpenAI’s Assistants API.
- Practice the canonical question above. The shape will repeat across companies; the depth varies by interviewer.
Frequently Asked Questions
Is this question asked outside AI labs?
Increasingly yes at AI-native startups, AI-product teams at FAANG, and senior+ ML engineer roles broadly. Pure FAANG product engineering interviews still rarely ask it.
Do I need to have built an agent platform to answer well?
Helpful but not required. Strong general system design plus familiarity with the specific subdomain considerations (tool orchestration, sandboxing, long traces) is enough.
How does this differ from designing a workflow engine?
Substantial overlap. Agent infrastructure is workflow engine + LLM-specific concerns (token budgets, prompt management, tool call grounding, hallucination handling, cost-per-trace).
What about agents-as-a-service vs internal-only agents?
The customer-facing platform problem is harder because of multi-tenancy, billing, and adversarial input. Internal agent infrastructure is simpler in those dimensions but still has all the orchestration and observability questions.
Are there reference implementations I can study?
LangChain (LCEL + LangGraph), Temporal-based agent implementations, Anthropic’s published Claude Code architecture details, OpenAI’s Assistants API documentation. Studying open-source agent frameworks gives concrete reference architectures.