The fastest way to fail this design is to send a one-time passcode and a promotional email through the same queue. Under normal load nobody notices. Then a marketing team schedules a blast to forty million users at 9am, the queue fills with two hours of low-value email, and the passcode a user needs to log in shows up four minutes later. That scenario is what most notification-system interviews are really testing: can you keep the message that matters ahead of the message that doesn’t.
Scope it out loud before drawing anything. You’re building a service other teams call to reach users across push, email, SMS, and in-app channels. It has to respect per-user preferences, survive a ten-times traffic spike, drop duplicates, and fail without losing the notifications people actually care about. Assume roughly a billion sends a day, about 12,000 per second on average and north of 100,000 per second during a spike. In-app read volume is separate and usually higher.
The follow-ups are predictable, and hearing them phrased the way they land in the room helps you prep the real answer:
- “A user got the same push twice. Where should dedup have caught it?”
- “Marketing wants to reach eighty million users at noon. What happens to password-reset codes during that window?”
- “How do you know a notification reached the device rather than only the provider?”
- “A device token is dead. What does your system do the next ten times someone tries to reach it?”
The write path, and why it returns 202
A service that wants to notify a user posts to your API: a recipient, a template id, some variables, maybe a channel hint. The API validates the request, assigns a notification id, writes almost nothing synchronously, and returns 202 Accepted. It does not send anything inline. Call FCM or Twilio inside the request handler and you’ve tied the caller’s latency to a third party you don’t control, so a Twilio hiccup surfaces as errors in someone else’s checkout flow.
Behind the 202, the notification lands on a queue. Kafka is the usual answer, and the reason to name it is partitioning. Key by user id so one person’s notifications stay ordered, and run separate topics per priority class. That second part is the whole game. One topic for transactional traffic (passcodes, password resets, payment receipts), a different topic for bulk and marketing. Workers on the transactional topic are provisioned to stay near-empty; the marketing topic can build a backlog without ever touching them.
Routing: preferences, quiet hours, and duplicates
A router pulls a notification and decides whether it should go out at all, and on which channel. User preferences live here. Someone who turned off marketing push still gets security alerts, because you classify notifications by type and check the preference for that type, not a global on/off switch. Quiet hours belong here too. A 2am promotional push in the user’s local timezone gets held or dropped, while a fraud alert ignores quiet hours by design.
Deduplication happens before dispatch. Two upstream services both firing an “order shipped” event for the same shipment is common, and the fix is an idempotency key computed from user, event, and channel, held in Redis with a TTL of a few hours. First writer wins, later duplicates die. The same key protects against your own retries firing twice.
The stored state is smaller than people expect. A notifications table that is append-heavy and partitioned by time, with old rows aged out, carries the per-message lifecycle. A small, cacheable preferences store answers the routing question. Templates live separately so a copy change doesn’t require a deploy of the pipeline.
Dispatch, and the exactly-once illusion
Channel dispatchers are thin adapters, one per provider: FCM and APNs for push, SendGrid or Amazon SES for email, Twilio or SNS for SMS, a WebSocket gateway for in-app. Each speaks a different protocol and fails in a different way, so keep them isolated. When SES starts throttling, email backs up while push and SMS keep moving.
You cannot promise exactly-once delivery across a network you don’t own. What you can promise is at-least-once with idempotent dispatch, which users experience as exactly-once nearly all the time. Send, record that you sent, and if the provider call times out with no clear answer, the idempotency key stops the retry from becoming a duplicate. Interviewers push here, so say it plainly: exactly-once is a marketing phrase, and the real design is at-least-once plus dedup.
Fan-out for the broadcast case
“Notify these three users” and “notify all fifty million users” are different problems wearing the same API. Expanding fifty million recipients inside one request is a non-starter. The API accepts a segment or audience id and returns immediately, then a fan-out worker reads the audience in batches and turns each batch into ordinary per-user notifications on the bulk topic. That’s fan-out-on-write, and it wins for anything you want delivered soon.
Fan-out-on-read shows up in the in-app channel, where a user opens the app and pulls their feed. For that read-heavy surface you store notifications per user and let the client fetch, rather than pushing every feed update through the pipeline. Most real systems run both: write-side fan-out for the channels that leave the building, read-side for the in-app feed.
Retries, dead letters, and what actually got delivered
Providers fail in two ways that need different handling. A 500 or a timeout is transient, so retry with exponential backoff and jitter; the jitter keeps a recovering provider from getting flattened by a synchronized wave of retries. A hard rejection, like an invalid phone number or an unregistered device token, is permanent. Don’t retry it, mark it failed, and prune the dead token so you stop paying to send into nothing. After a bounded number of transient retries a notification goes to a dead-letter queue for inspection instead of looping forever.
Sent and delivered are different events, and strong answers separate them. Handing a push to APNs tells you it was accepted, not that it reached the phone. Delivery and read confirmations arrive asynchronously through provider webhooks and client callbacks, and you feed those back to update state. A clean lifecycle runs created, queued, sent, delivered, read, or failed, with a timestamp on each transition. That history is what lets you answer the user who swears the alert never came.
Picking the right channel for the message
Cost and latency differ by an order of magnitude across channels, and the wrong choice is how you either annoy users or burn money. Naming the tradeoffs concretely is part of a good answer.
| Channel | Typical providers | Delivery latency | Delivery confirmation | Rough cost per message | Best fit |
|---|---|---|---|---|---|
| Mobile push | APNs, Firebase Cloud Messaging | Seconds | Async receipt, not fully reliable | Effectively free | Re-engagement and timely app alerts |
| SendGrid, Amazon SES | Seconds to minutes | Webhooks for delivered, bounce, and open | Fractions of a cent | Receipts, digests, and long-form content | |
| SMS | Twilio, Amazon SNS | Seconds | Carrier delivery receipt | Cents, varies sharply by country | One-time passcodes and urgent alerts |
| In-app | Your WebSocket gateway and feed store | Instant if online, else on next fetch | Read state tracked directly | Effectively free | Activity feed and non-urgent updates |
The number worth committing to is transactional delivery latency at p99, not average throughput. Average throughput is easy, and every candidate quotes it; the interesting failure is the passcode stuck at four minutes while the mean looks fine. Protect it with the priority split, a per-user rate limit that caps how many notifications one person can get in an hour so a buggy upstream loop can’t fire two hundred pushes at someone, and batching that rolls “Alice, Bob, and three others liked your photo” into one message instead of five. Skip the rate limit and the fatigue problem quietly turns into an uninstall problem, which no amount of clever queuing will fix.
Keep sharpening your system design:
