The fastest way to stall in a rate limiter interview is to answer “token bucket” in the first ten seconds. The algorithm is the part everyone memorizes, and the interviewer knows it. What they want to see is whether you understand where the limiter sits, what state it keeps, and what happens to that state once you have more than one server. Plenty of candidates get the counting math right and then wave a hand over the distributed part, which is the exact part the question exists to probe.
A typical prompt: an API serving a few hundred million requests a day, and the product team wants free-tier users capped at 100 requests per minute while paid users get 1,000. That is enough detail to start drawing, and enough ambiguity that you should ask a few questions first.
Scope it before you draw anything
The first three or four minutes are for narrowing what “rate limiter” means in this problem. The answers move the whole design, so pull them out early rather than discovering them halfway through. Good questions to ask out loud:
- What is the limit keyed on: user ID, API key, source IP, or a pair like user plus endpoint?
- Is the cap global across the fleet, or per region?
- Do we reject over-limit traffic with an error, or queue and delay it?
- How exact does the count need to be: is briefly allowing 105 requests against a 100-per-minute cap a real bug, or a rounding error nobody notices?
- What matters more, never blocking a legitimate user or never letting an abuser slip through?
That last question carries more weight than it looks. A rate limiter is an availability tradeoff wearing a correctness costume. If the store holding your counters goes down, do you let every request through (fail open) or block everything (fail closed)? A login endpoint under a credential-stuffing attack and an internal analytics API answer that very differently, and saying so tells the interviewer you have run one of these in production.
Where the limiter lives
Before counters and algorithms, decide where the check happens. Putting it in the client is a non-starter for anything security-adjacent, since a client can lie about its own request count, though client-side limits still help as a courtesy to avoid wasting round trips.
The common answer is a shared layer that every request already passes through: an API gateway, a reverse proxy like Envoy or NGINX, or a load balancer with a rate-limiting module. One place to configure, one place to enforce, and the rejected traffic never reaches your application servers. The alternative is middleware inside each service, which gives you limits that understand business context (this specific endpoint, this account’s plan) at the cost of duplicating the logic and the counter access across every service. Large shops often run both: a coarse gateway limit to shed obvious floods, and finer per-endpoint limits deeper in.
There is also the dedicated rate-limiter service or sidecar, where the gateway makes a fast call to a separate component that owns all the counting. It centralizes the policy and the state, and it adds a network hop on the hot path. Mention it, note the latency cost, and move on unless the interviewer bites.
The algorithm is a ninety-second decision
This is the memorized part, so spend little time and show you know the tradeoffs rather than reciting definitions. Fixed-window counters are trivial and let a caller sneak nearly double the limit across a window boundary. Sliding-window counters fix most of that edge burst with two integers and a weighted blend. Token bucket is the one to reach for when you want to permit controlled bursts, which most public APIs do, because real traffic is bursty and a hard per-second wall annoys well-behaved clients.
| Algorithm | What it stores per key | Memory per key | Burst behavior | Typical use |
|---|---|---|---|---|
| Fixed window counter | One count per key per time window, reset on the boundary | 1 integer | Allows up to 2x the limit across a window edge | Simple internal or low-stakes APIs |
| Sliding window log | A timestamp for every request in the window | One entry per request | Exact, no edge burst | Low-volume limits that must be precise |
| Sliding window counter | Current and previous window counts, weighted by elapsed time | 2 integers | Smooths the edge burst, slightly approximate | Most production API gateways |
| Token bucket | Current token count and last refill timestamp | 2 numbers | Permits bursts up to the bucket size, then the refill rate | Public APIs, network shaping |
| Leaky bucket | A queue drained at a fixed rate | Queue length | No burst, smooth constant output | Shaping outbound calls to a fragile dependency |
The real question is where the counter lives
Everything above works on one machine with an in-memory map. The interview turns on what breaks when you have fifty gateway nodes behind a load balancer.
Say a user is allowed 100 requests per minute and their traffic gets spread evenly across two gateways. Each node keeps its own counter, sees 50 requests, and decides the user is nowhere near the limit. The user sails through at 200. Round-robin balancing guarantees this, and even sticky sessions only delay it, because sessions expire and nodes fail. Local counters do not compose.
The standard fix is a shared store that every node reads and writes, and Redis is the usual answer because it is in-memory, single-threaded per shard, and ships atomic operations that do the counting for you. A fixed-window check becomes one command:
-- KEYS[1] = "ratelimit:user:123:minute"
-- ARGV[1] = limit, ARGV[2] = window seconds
local current = redis.call("INCR", KEYS[1])
if current == 1 then
redis.call("EXPIRE", KEYS[1], ARGV[2])
end
if current > tonumber(ARGV[1]) then
return 0 -- reject
end
return 1 -- allowWrapping the increment and the expiry in one Lua script matters. If you run INCR and EXPIRE as two separate calls and the process dies between them, you get a key with no TTL that counts forever and locks the user out until someone notices. Token bucket is the same story with more state: you read the token count and last-refill time, compute how many tokens to add for the elapsed time, decrement, and write back. Done as read-modify-write across the network, two requests can both read “1 token left” and both spend it. The read, the refill math, and the decrement have to be one atomic step, which is why the logic goes into a Lua script or a Redis function rather than your application code.
Making it fast without lying about the count
A Redis call on every single request buys you an accurate global count and costs you a network round trip plus a hard dependency on Redis being up. At a few hundred million requests a day that is real latency and a real failure domain. This is where you show judgment about how much accuracy the problem actually needs.
One pattern is a local token cache per node. Each gateway leases a slice of the budget from Redis, say 20 tokens at a time, serves requests from that local slice at memory speed, and goes back for more when it runs low. Counts drift slightly around the boundaries, and you trade some precision for a large drop in Redis traffic and latency. For a public API where the difference between 100 and 103 requests is meaningless, that trade is easy. For a limit that guards a paid metering boundary, it is not.
The other lever is the fail-open versus fail-closed decision from earlier, now with teeth. If Redis is unreachable, a fail-open limiter serves the request and logs the miss, keeping your API available at the cost of a temporary hole in enforcement. A fail-closed limiter returns errors until the store recovers. Name which one you would pick for the endpoint in the prompt and why, because the interviewer is listening for exactly that reasoning.
What the client should get back
A rejected request returns HTTP 429, Too Many Requests. On its own that teaches the client nothing, so send the headers that let a well-behaved caller back off cleanly: Retry-After with the seconds to wait, plus X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset so clients can pace themselves before they ever hit the wall. A limiter that just slams the door produces retry storms, where rejected clients immediately hammer you again and make the overload worse. The headers are cheap and they change client behavior, so mention them even if the interviewer never asks.
Where the follow-ups go
Once the base design holds, interviewers push on the parts that get genuinely hard at scale. Hot keys are the favorite: a single popular API key or a viral endpoint routes all its counting to one Redis shard and melts it while the rest of the cluster sits idle. Answers involve sharding the counter itself, adding a local absorbing layer in front, or accepting approximation for the hottest keys. Multi-region is the other one, since a globally consistent counter across continents means cross-region latency on every request, and most teams settle for per-region limits with an accepted global overshoot rather than pay that cost. If you have already said out loud that you know how much accuracy the limit needs, that conversation stays short, because you decided the answer back in the first five minutes.
Keep sharpening your system design:
