“Design just-in-time production access for 2,000 engineers” is a good security system design prompt because the obvious answer is wrong. The obvious answer, a Slack bot that grants an IAM role for an hour, becomes the most valuable thing in the company to compromise, it trains approvers to click yes without reading, and it falls over the night the identity provider does.
Other security system design interview questions in the same family:
- Design a secrets manager for a microservice fleet.
- Design a secure file-sharing service for enterprise customers.
- How would you harden this Kubernetes cluster?
- Design an authentication service for a web app.
- Design IAM for employees and external partners.
Several have low-level designs here, linked below; other format pages are on the security engineer interview guides hub. For a 60-minute round, budget about eight minutes for questions and requirements, five for assets and adversaries, ten for components and trust boundaries, and twenty for the threat-driven decisions. That’s 43 minutes; in a 45-minute slot, cut each block by a third. The failure table and rollout plan below are follow-up material for when the interviewer asks.
Turn clarifying questions into security requirements
Say which requirement each answer creates. Interviewers credit requirements you state, not ones that stayed in your head. Ask these, with an assumption ready in case the question comes back:
- What do engineers reach today? Assume SSH to 30,000 hosts, kubectl on 40 clusters, 150 AWS accounts and production Postgres holding customer data.
- How much access is standing? Assume 600 people hold permanent admin somewhere.
- How fast does on-call need in? Assume 250 engineers on rotation and two minutes at 3am.
- What has the company promised outsiders? Assume an external audit and contracts saying every human touch of customer data is attributable.
- What exists already? Assume SSO, managed laptops and hardware security keys for staff.
Those answers become six requirements, written in a corner of the board. No human holds standing production credentials. Every grant names a person, resource, privilege level, reason (an incident or ticket ID) and expiry, one hour by default and eight at most. Approval scales with risk, from automatic for low-risk reads to two approvers for customer-data writes. Access works when the identity provider or the access service is down. Every session is attributable to one human with its commands logged, and the riskiest sessions can be replayed in full. And the access system must be a harder target than the production it guards, because centralizing access builds the thing an attacker most wants to own.
Assets and adversaries, ranked instead of listed
Assets first, roughly by value: customer data in production databases; the cloud control plane, where one IAM change grants everything else; the access system’s signing keys; production hosts; and the audit trail, because an attacker who can edit it can hide the rest. Then adversaries, likeliest first:
- An outsider with a phished or purchased engineer password. The design is built around this one.
- Malware on an engineer’s laptop, riding an authenticated session.
- An insider on call who takes more than the incident needs.
- Whoever compromises the access system itself: broker, issuers, proxy or their deploy pipeline.
The first isn’t hypothetical. Uber’s account of its September 2022 breach describes an attacker who most likely bought a contractor’s password, was blocked by repeated two-factor prompts, and got in when the contractor accepted one. Press coverage reported the attacker then finding a PowerShell script with admin credentials for the platform Uber used to manage login secrets. Push fatigue at the door and a secrets platform as the prize: a JIT design answers both.
Name what’s out of scope: machine identities and service-to-service auth, the production deploy pipeline, supply-chain compromise and customer-facing login.
The components to draw before any arrows
laptop (managed device, hardware key)
-> IdP: SSO + WebAuthn <- device posture from MDM
-> access broker: requests + policy <-> approvals in chat or web
-> issuers: SSH CA (HSM) / AWS STS / DB users / k8s OIDC
-> access proxy: terminates and records sessions, AWS egress
-> production: hosts, clusters, databases, AWS accounts
broker, issuers, proxy, hosts -> audit sink (own account, append-only)
break-glass: sealed, split custody, offline (no arrows until used)The broker evaluates policy, routes approvals and records outcomes. The issuers are deliberately narrow: an SSH certificate authority backed by an HSM, an AWS issuer that assumes predefined roles through STS, a database issuer that creates per-session users, and Kubernetes accepting short-lived OIDC tokens. The access proxy terminates and records every SSH and database session and carries AWS traffic out. The audit sink sits in a separate account everything can write to and nothing can delete from.
Splitting decision from issuance is what makes a broker compromise survivable. Each issuer checks the requester’s own fresh IdP token, including the recorded hardware-key step-up, plus signed approvals wherever the tiers below require them. Incident overrides have no human approval, so the issuer re-checks incident state itself. A compromised broker can’t mint a grant for someone who didn’t ask. Issuers also enforce their own hard limits (no certificate past eight hours, no role outside an allowlist).
Trust boundaries, and one request walked across them
Draw the boundaries, then walk one request across them. Priya is on call for payments, gets paged at 3:10am for INC-4821, and needs read access to the payments database plus SSH to its API hosts.
Boundary one separates her untrusted laptop and network from corporate identity. What crosses is a WebAuthn assertion from her hardware key plus a device-posture claim, so a phished password alone doesn’t get across.
Boundary two separates identity from authorization. The broker receives her fresh identity token and a request: payments-db, read, reason INC-4821, one hour. Customer-data reads are tier two, but the incident override applies. The incident system confirms INC-4821 is open on payments and paged Priya’s rotation, so the grant auto-approves and posts to the team channel.
Boundary three is the signing domain. The SSH CA issues Priya a one-hour certificate for a diagnostic-only principal, pinned by force-command to read-only tools so it counts as an incident read, and the database issuer creates a read-only user that expires with the grant. OpenSSH handles validity intervals, principals and options like source-address and force-command natively, per the ssh-keygen certificate documentation.
Boundary four is the edge of production, and the proxy sits on it. It terminates Priya’s SSH session, which it must do to record it, then opens the host leg with a per-session certificate restricted by source-address to the proxy’s addresses. Network policy admits SSH, database and Kubernetes API connections from the proxy and nothing else. When the hour ends, every credential involved stops working without anyone revoking it.
AWS needs one more step, because STS credentials work from anywhere by default. Route CLI and console traffic out through the proxy, and give every role the issuer can assume a condition on aws:SourceVpce or aws:SourceIp, with the usual carve-out for calls AWS services make on your behalf. Credentials copied off a laptop then fail everywhere else.
The decisions each threat forces
Each control below answers a named adversary from the list.
Authentication: hardware keys, because push approvals get spammed
Require a hardware key or platform authenticator at grant time, on top of the morning login. Push approval is what a phished-password attacker beats by sending prompts until someone taps yes. Number matching helps, but WebAuthn is better because the signature is bound to the site’s origin, so a phishing proxy can’t relay it. Step-up per grant also blunts laptop malware, since a session stolen at 9am can’t mint a certificate at 3am. Token mechanics (refresh rotation, revocation, TOTP) are in the user authentication LLD.
Authorization: approval rules people won’t rubber-stamp
Use three tiers. Tier one, read-only on non-sensitive systems, auto-approves for the owning team. Tier two, any write or any customer-data read, needs one approver from the owning team other than the requester, with one override: tier-two reads auto-approve when the requester is on the paged rotation for an open incident on that service, with the channel post and row limits on the database session as compensating controls. Tier three, customer-data writes or any IAM change, needs two approvers, one outside the requester’s management chain. Nobody approves their own request or one that widens their own access.
Policy lives in code, and the broker evaluates attributes (team, rotation, incident state, sensitivity) on top of role-based defaults. Decision and enforcement points are in the access control system design page, and the role model in the auth service LLD. Interviewers push on the human side: an approver who sees forty requests a day approves all forty. Make tier one cover most real work and watch approval time. A median under three seconds means nobody is reading.
Secrets: credentials that expire on their own
The best secret here is useless in an hour. Certificates, STS sessions and per-session database users all expire on their own, so revocation becomes a timeout instead of an emergency. What stays static is small: the CA signing key, held in an HSM that signs but never exports it (the HSM key custody in the certificate authority LLD carries over), and the issuers’ own service credentials, which get envelope encryption, rotation and workload identity as in the secret manager design. A fleet still on static SSH keys needs the rotation mechanics in the SSH key rotation LLD during the migration. And per Uber, the access platform’s admin credential can’t live in a script or CI variable; its operators request JIT access to it like everyone else.
Isolation: the broker, issuers and proxy are crown jewels
The access system gets its own cloud accounts, a small admin group, and two-person-reviewed deploys from a pipeline production teams can’t push to. The broker holds no production credentials. The proxy needs the most care, because terminating sessions means it holds credentials for every live connection. Keep those per-session and short-lived, never a static database password, and reconcile host-side sshd and database logs against the proxy’s own logs so a proxy that lies about sessions gets caught. Proxy-only access is enforced in network policy, so no team can quietly open port 22 to the VPN.
Audit: a record the attacker can’t edit
Requests, decisions, issuances, sessions and commands go to an append-only store with object-lock retention and a hash chain so edits and gaps show. The audit log system LLD has the schema. What the interviewer wants is reconciliation: issuances against user checks and approvals, proxy sessions against host logs. Anything in one record but not the other is an alert.
Abuse and rate limits on the access path
Cap pending requests per person, rate-limit issuance per identity, and validate incident and ticket IDs against the systems that own them, so a request citing a closed incident gets denied and flagged. Throttle repeats after a denial, because approval spam is push fatigue from the inside. The counting is an ordinary rate limiter design. The security choices are the key (the human, not the IP) and the response at the limit (deny and alert, never queue).
Failure modes, and how you’d notice each one
Follow-up material, for when the interviewer starts breaking boxes.
| Failure in a just-in-time access system | What the attacker or outage gets | How you detect it | Design response |
|---|---|---|---|
| Identity provider, broker or issuer outage | No new grants; on-call is locked out and teams lobby for standing access | Synthetic end-to-end grant every minute | Active-active issuers; issued credentials work until expiry; break-glass for IdP loss |
| Access broker or credential issuer compromised | Grants for people who never asked, or approvals that never happened | An issuance with no matching user-token check or approval record | Issuers verify the requester’s fresh IdP token and signed approvals themselves |
| SSH certificate authority signing API misused | Certificates for any principal, signed by the real non-exportable key | Access proxy or host sees a certificate serial with no issuance record | Signing callable only by the SSH issuer; CA rotation; short lifetimes |
| Access proxy compromised | Live sessions, their per-session credentials, forged recordings | Host sshd and database logs that don’t reconcile with the proxy’s logs | Per-session credentials only; proxy in its own account; rebuild from image |
| Approvers rubber-stamping requests | The second-approver check on writes and customer-data reads becomes a formality | Median approval time; per-approver approval rate near 100% | Fewer approvals, plainer request text, periodic access review |
| Valid grant misused by laptop malware or an insider | Real access put to the wrong use | Commands, rows or bytes per session against the incident’s baseline | Step-up per grant; proxy-bound SSH and AWS credentials; row limits; session kill |
| Audit pipeline silently dropping events | Actions with no record | Heartbeats and sequence gaps in the hash chain | Proxy buffers records; only non-sensitive and incident reads continue; customer-data writes and IAM changes fail closed |
Break-glass needs thirty seconds, because candidates either forget it or design a backdoor. The credentials are sealed, split between two or three holders and usable only from known devices. Any use pages security, rotation follows the incident, and the procedure gets rehearsed quarterly.
Trade-offs you should defend out loud
Speed against control on call is the biggest trade-off. The incident override means a phished on-call engineer reads customer data with no second human looking. Take the trade and say why: without it, on-call keeps a standing key “just in case,” while the override is still bounded by the hardware key, the incident check, the expiry, row limits and a channel post. Writes never auto-approve.
The proxy is a choke point, an extra hop and a high-value target, but it’s the only place to record sessions and enforce the network rule. Scale it out and accept the hop.
Fail open or fail closed is the interviewer favorite, and the answer is per tier. When the audit sink is unreachable the proxy buffers records locally: tier one keeps going, tier two allows only incident-override reads and pages security, and tier three stops. “Always fail closed” sounds secure and gets the system ripped out after its first bad outage.
Keystroke capture also records secrets pasted into terminals, so log commands everywhere and keep replayable full sessions for tier three. Access proxies already exist, commercial and open source; offering to evaluate them while owning the policy beats building a CA to prove you can.
How the round differs from a regular system design interview
The skeleton matches the rounds in the system design interview guides: requirements, components, one area in depth, trade-offs. What gets scored changes. On an access prompt, 2,000 engineers making a few requests a day is a rounding error and nobody asks about QPS.
Requirements include adversaries, and “who attacks this” reshapes the design the way the read-to-write ratio does for a news feed. Depth goes where trust concentrates, the CA or the approval policy, rather than where load is. And failure means compromise. “The broker is owned, now what?” gets answered with what the issuers still verify on their own and how reconciliation catches the forged grants, not with more replicas.
The round often opens like a threat model. The threat modeling interview walkthrough builds a full table on the STRIDE categories Microsoft documents, and the secure code review interview page covers the round where you get code instead of a whiteboard.
How the security system design round is graded
We couldn’t find a public Meta source on this interview, so nothing here describes Meta’s format. What Meta has published is its production SSH architecture: no direct SSH to production, bastion hosts, and certificates listing each engineer’s allowed principals (Marlon Dutra, “Scalable and secure access with SSH,” 2016). That’s the same certificate-and-bastion shape as the design above. The Meta interview guide covers the company’s general loop.
Scoping comes first: did you ask what’s in and out, and come back to the requirements. Then threat judgment, whether you ranked realistic adversaries or recited a list. Then control fit, whether each control answers a stated threat and you know what it doesn’t stop. Then depth, two levels down on one component when pushed. Then operability, whether on-call would use the thing at 3am. Communication runs through all of it: name the trade-off before you’re asked. Seniority shows in the last two, and a senior answer also explains how 2,000 people who like their standing access get moved off it.
Where candidates lose this round
The most common miss is opening with controls. “mTLS, encryption at rest and a WAF” in minute two says you’re pattern-matching, because you haven’t said what you’re protecting or from whom.
Next is the unranked threat list. Twenty equal-weight threats lose to three ranked ones with reasons, a point the security engineer loop overview makes too.
Then there’s the design that’s secure and unusable. If incident access takes ten minutes and three approvals, on-call hides a standing key somewhere, and you’ve made things worse while looking better on paper.
Forgetting that the access system is a target, skipping break-glass, and treating audit as a bucket nobody watches all mark someone who has read about these systems rather than built one.
If you have time left, spend it on rollout, because that’s where real JIT projects die. Run the broker in shadow mode and log who uses standing access, for what and how often. Move the heaviest users first, since they find the rough edges, then remove standing access team by team against a dated cutoff. The number leadership should watch is standing production credentials remaining, and the design only pays for itself the day it hits zero.
Keep sharpening your system design:
