Nothing sinks a system design loop faster than using a word you can’t defend. You say “we’ll shard the database” and the interviewer immediately wants to know what you shard on, what happens to a query that spans shards, and how you rebalance when one shard runs hot. The terms below are the shorthand interviewers use to probe how deep your understanding actually goes. Drop one loosely and you invite exactly the follow-up you’re least ready for.

Each entry is short: what the word means, the tradeoff attached to it, and the follow-up it triggers.

Latency, throughput, and load

p99 (tail latency)

The p99 is the latency value that 99% of requests come in under. If your p99 read latency is 300 ms, one request in a hundred takes longer than that. Averages hide pain: a service can post a 20 ms mean while its p99 sits at 900 ms because a slow tail is dragging real users. Interviewers ask for p99, and sometimes p999 (the 99.9th percentile), because that tail is what a user with ten downstream calls actually feels. A page that fans out to 20 services hits its slowest backend’s tail almost every time.

Throughput

Throughput is how much work the system finishes per unit time: requests per second, messages per second, rows written per second. It sits on a different axis from latency. You can raise throughput by batching and adding parallelism while individual requests get slower. When someone asks you to “design for 50k writes per second,” they’ve handed you a throughput target, not a latency one.

Backpressure

Backpressure is a slow consumer telling a fast producer to ease off. Without it, a producer that outruns its consumer fills queues until memory runs out or messages get dropped. Real systems apply it by bounding queue sizes, blocking the producer, or returning an explicit signal to slow down (an HTTP 429, a full channel, a paused Kafka consumer). If your design has a queue in it, expect “what happens when the consumer can’t keep up?” and backpressure is the word they want to hear.

Load shedding

When a system is past its limit, load shedding means dropping some work on purpose so the rest survives. That might be rejecting low-priority requests, returning cached or degraded responses, or failing fast with a 503 instead of letting everything time out. A server crushed to a crawl serves nobody, so refusing 10% of traffic keeps the other 90% healthy. It pairs with rate limiting, which caps load before it arrives rather than after.

Splitting data across machines

Sharding and the partition key

Sharding splits one dataset across many machines so no single box holds all the data or all the traffic. The partition key decides which shard a row lands on: user_id, tenant_id, a hash of the primary key. Pick it well and load spreads evenly; pick it badly and you get a hotspot, one shard swamped while the rest sit idle. The classic trap is sharding a chat app by user when a handful of celebrity accounts generate most of the messages.

Consistent hashing

Consistent hashing lets you add or remove a shard without reshuffling almost everything. Plain modulo hashing (hash(key) % N) remaps nearly every key when N changes; consistent hashing places servers and keys on a ring so only the keys near the changed node move. It’s why systems like Cassandra and DynamoDB can grow a cluster without a full data migration. Expect it the moment you say “add more nodes” out loud.

Replication

Replication keeps copies of the same data on multiple machines, for durability and for read scaling. Writes go to a leader and flow out to followers, or in a leaderless setup any replica accepts writes and they reconcile later. The tension is lag: a follower a few hundred milliseconds behind can serve a read that’s missing a write you just made. That gap is where consistency models come in.

Denormalization

Denormalization means storing redundant copies of data so a read doesn’t have to join across tables or services. You trade write complexity and storage for fast reads, precomputing the shape the read path wants. A feed that stores each post already copied into follower inboxes is denormalized: the write does more work so the read is a single lookup.

Consistency and coordination

CAP theorem

CAP says that when a network partition splits your nodes, you choose between consistency (every read sees the latest write) and availability (every request still gets an answer). You don’t get both during the partition, so real systems pick a default: a payment ledger leans CP and refuses writes it can’t confirm, a shopping cart leans AP and accepts writes it will reconcile afterward. The common mistake is treating CAP as a permanent three-way choice. The tradeoff only bites during a partition; the rest of the time you’re trading latency against consistency, which is what PACELC adds.

Strong versus eventual consistency

Strong consistency means a read always reflects the most recent write, as if there were a single copy of the data. Eventual consistency means replicas converge given enough time, so a read right after a write can return stale data. Most large systems run eventually consistent by default and reserve strong guarantees for the parts that need them, like account balances. Read-your-writes is the middle ground a user notices: they should at least see their own edits immediately, even if others see them a beat later.

Quorum

A quorum is the minimum number of replicas that must acknowledge an operation for it to count. With N replicas, if writes need W acks and reads need R responses, then W + R > N guarantees a read overlaps at least one node holding the latest write. The usual setup is N=3, W=2, R=2, which tolerates one node down while staying consistent. Tuning W and R up or down is how you slide between faster-but-staler and slower-but-safer.

Linearizability

Linearizability is the strongest single-object guarantee: once a write completes, every later read by wall-clock time sees it, and the system behaves as if operations ran in one order everyone agrees on. It’s what people usually mean when they say “strongly consistent.” It costs coordination, which costs latency, so you apply it to the few operations that truly need it, like acquiring a lock or incrementing a counter that must never double-count.

Staying up when things fail

Idempotency

An idempotent operation produces the same result whether you run it once or five times. It’s the property that makes retries safe: if a payment request times out and the client retries, an idempotent API charges the card once, not twice. The usual implementation is an idempotency key the client sends, which the server records so it can recognize and ignore duplicates. Any time your design retries anything, expect “how do you avoid double-processing?” and this is the answer.

Delivery guarantees

Message systems promise one of three things about how often a message arrives. At-most-once can drop messages but never duplicates them; at-least-once never loses a message but can deliver it twice; exactly-once is the ideal that’s expensive and usually approximated with at-least-once delivery plus idempotent consumers. Most production pipelines run at-least-once and make the consumer idempotent, because true exactly-once across a network is slow and hard to guarantee end to end. The table lays out the tradeoff.

Delivery guarantee	What it promises	How it is usually built	Failure mode to watch
At-most-once	Each message delivered zero or one time, never twice	Fire-and-forget, no retries or acknowledgements	Silent message loss when a failure hits
At-least-once	Each message delivered, possibly more than once	Retries until the consumer acknowledges receipt	Duplicates the consumer has to handle
Exactly-once (effective)	Each message takes effect a single time	At-least-once delivery plus an idempotent consumer or dedup key	Cost and complexity; rarely guaranteed end to end

Circuit breaker

A circuit breaker stops calling a downstream service that’s already failing so you don’t pile retries onto a system that’s down. After a threshold of failures it trips and fails fast for a cooldown window, then lets a trickle of requests through to test recovery before closing again. Without one, a single slow dependency can exhaust every thread waiting on it and drag down services that had nothing wrong with them. That chain reaction is a cascading failure, and the breaker exists to stop it.

Dead letter queue

A dead letter queue is where messages go after they’ve failed processing too many times. Instead of retrying a poison message forever (a malformed payload, a record that trips a bug), the system parks it in a separate queue for a human or a later job to inspect. That keeps one bad message from blocking the whole pipeline. When you say “we retry failed messages,” the natural follow-up is “and when they keep failing?” — the dead letter queue is your answer.

Caching, and why it’s hard

Cache hit ratio and TTL

The hit ratio is the share of reads served from cache instead of the source of truth; 95% means only one read in twenty reaches the database. TTL, or time to live, is how long an entry stays valid before it’s evicted or refetched. Short TTLs keep data fresh but push more load to the origin; long TTLs cut load but serve staler data. Tuning those two against each other is most of what cache design actually is.

Thundering herd

A thundering herd is what happens when a popular cache entry expires and thousands of requests all miss at once, hammering the database in the same instant. The fixes are to let one request rebuild the entry while others wait or serve stale, to add small random jitter to TTLs so keys don’t expire together, or to refresh hot keys before they lapse. It’s a favorite follow-up after you casually say “we’ll cache it,” because it shows whether you’ve thought past the happy path.

The words for “how good is good enough”

SLA, SLO, and SLI

An SLI is the thing you measure, say the fraction of requests under 200 ms. An SLO is the target you hold yourself to internally, like 99.9% of requests under 200 ms this month. An SLA is the contractual promise to a customer with penalties attached, and it’s usually looser than the SLO so you keep some headroom. Mixing these up is a small tell that you haven’t run something in production, since the SLO is the number engineers actually work against.

Fan-out

Fan-out is one event triggering many downstream writes or calls. Fan-out on write pushes a new post into every follower’s feed at write time, making reads cheap and writes expensive; fan-out on read assembles the feed by querying everyone you follow at read time, cheap to write and costly to read. Most large feed systems run a hybrid, fanning out on write for normal users and on read for accounts with millions of followers. Naming which one you mean, and why, is half the answer to the timeline design question.

None of these words carry much weight on their own. What an interviewer is listening for is whether you reach for the right one at the right moment: quorum when you’re talking replicas, backpressure when you draw a queue, idempotency the instant you mention a retry. Learn them well enough that they turn up in your answer before anyone has to ask.