Question 1

How do you design a multi-channel notification dispatcher that supports push, email, SMS, and in-app?

Accepted Answer

Model the dispatch pipeline as: (1) a `notification_requests` intake API that accepts (user_id, event_type, payload) and writes to an intake queue; (2) a router service that loads user channel preferences and opted-in channels from a preferences store (Redis cache backed by Postgres), then fans out to per-channel queues (push_queue, email_queue, sms_queue, inapp_queue); (3) per-channel workers that call the respective provider (FCM, SES, Twilio, internal WebSocket server). Store per-channel worker output in a `notification_deliveries` table (notification_id, channel, status, provider_message_id, timestamp). Channel preference logic handles fallback: if push is not available (no device token), fall back to email. De-duplicate at intake using an idempotency key so upstream retries don't produce duplicate notifications.

Question 2

How do you implement priority queues for notifications so that critical alerts aren't delayed by bulk sends?

Accepted Answer

Assign each notification a priority (e.g., CRITICAL, HIGH, NORMAL, LOW) at intake based on event_type. Route to separate queues per priority (critical_push_queue, bulk_push_queue) rather than a single queue with priority headers — separate queues allow independent scaling of consumers and prevent head-of-line blocking. Allocate consumer thread pools proportionally: e.g., 8 workers on critical, 2 on bulk. For Kafka-backed systems, use separate topics per priority and assign more partitions + consumer instances to higher-priority topics. Apply rate limiting on the bulk tier (e.g., 1000/s per channel) without throttling critical. In the database, index `notification_requests` on (priority DESC, created_at ASC) for any polling-based workers. Monitor queue depth and consumer lag per priority tier as primary SLIs; alert if critical queue lag exceeds 5 seconds.

Question 3

How do you track delivery status across multiple providers and surface it to callers?

Accepted Answer

Each channel worker writes an initial `notification_deliveries` row with status='sent' and the provider's message ID immediately after a successful API call. Providers deliver status callbacks (FCM delivery receipts, SES SNS bounce/delivery events, Twilio webhooks) to a callback ingestion endpoint, which updates the delivery row to 'delivered' or 'failed' with a failure reason code. For providers that don't support callbacks (some SMS carriers), poll the provider status API with exponential backoff up to a maximum staleness window (e.g., 24h), then mark as 'unknown'. Expose a status API: GET /notifications/{id}/status returns a rollup across all channels. Implement a dead-letter queue for failed deliveries with a retry policy (3 attempts with exponential backoff); after exhausting retries, emit a `notification.failed` event for upstream alerting or fallback channel escalation.

Question 4

How do you prevent notification storms and implement user-level rate limiting?

Accepted Answer

Apply two layers of rate limiting. First, per-user frequency capping: use a sliding window counter in Redis (INCR + EXPIRE or a sorted set with timestamps) keyed by (user_id, channel, window). If a user has already received N notifications of a given priority in the window (e.g., 5 push/hour for NORMAL), suppress or defer the new notification and log the suppression. Second, per-channel global throughput limiting: token bucket in Redis (or a rate-limiting sidecar like Envoy) enforces provider SLA limits (e.g., FCM: 600K/min). For bulk campaigns that could produce millions of notifications simultaneously, use a scheduled-dispatch pattern: write all intended notifications to a `scheduled_notifications` table and drain them through a controlled worker at a rate that respects global limits, rather than enqueuing all at once. Alert on suppression rate spikes as a leading indicator of upstream event loop bugs.

Notification Delivery Service Low-Level Design: Multi-Channel Dispatch, Priority Queues, and Delivery Tracking

Notification Schema

Channel Dispatch Flow

Priority Queues

Push Notification Worker

Email and SMS Workers

In-App Notification Worker

Delivery Status Tracking

Retry Policy with Exponential Backoff

Deduplication and Analytics