# Monitoring and Alerting System: Low-Level Design

Source: https://www.techinterview.org/post/3233473342/lld-monitoring-alerting-system/
Updated: 2026-04-18 · techinterview.org

A monitoring and alerting system collects metrics from distributed services, stores them efficiently, evaluates alert conditions, and notifies on-call engineers. Building one correctly requires understanding time-series data, alert evaluation semantics, and the operational properties of the pipeline itself — the monitoring system must be more reliable than the services it monitors.

## Metrics Collection

### Push vs. Pull

Pull-based (Prometheus model): the metrics server scrapes each service endpoint on a schedule. Services expose a /metrics endpoint; the server controls scrape frequency. Advantages: scrape failures are visible to the metrics server, no client-side buffering needed, service discovery is centralized. Pull works well in static infrastructure with known targets.

Push-based (StatsD, InfluxDB model): services push metrics to a collection agent. Better for dynamic environments (containers, lambdas), short-lived jobs, and when services can't expose an HTTP endpoint. Push requires handling backpressure — if the collection agent is slow, services must buffer or drop metrics.

### Metric Types

Counter: monotonically increasing (total requests, errors). Rate of change computed by the time-series database. Gauge: point-in-time value (current queue depth, active connections). Histogram: distribution of values (request latency buckets); enables percentile computation. Summary: pre-computed quantiles computed on the client (less flexible than histograms, cannot be aggregated across instances).

## Time-Series Storage

Time-series data (timestamp, value, labels) has specific access patterns: append-only writes, time-range reads, aggregation over time windows. Dedicated time-series databases (Prometheus TSDB, InfluxDB, VictoriaMetrics) compress sequential timestamps and float values efficiently (delta-of-delta encoding, Gorilla compression). A 16-byte float timestamp pair compresses to 1.37 bytes on average with Gorilla compression.

Retention strategy: store high-resolution data (15s scrape interval) for 15 days; downsample to 1-minute resolution for 90 days; downsample to 1-hour resolution for 2 years. This balances storage cost against the need for historical trend analysis. Downsampling aggregates: min, max, avg, sum over the interval.

## Alert Evaluation

### Threshold-Based Alerts

The simplest form: fire when a metric crosses a static threshold for a sustained duration. Example: alert if error_rate > 5% for 5 minutes. The sustained duration requirement ("for 5 minutes") reduces false positives from transient spikes. Trade-off: longer duration means slower detection. For critical metrics (disk full), use shorter durations; for noisy metrics, use longer ones.

### Anomaly Detection

Static thresholds miss seasonal patterns — a 10% error rate at 3am may be anomalous while 10% at noon may be normal. Anomaly-based alerts compare current values against a baseline (same time last week, rolling average ± N standard deviations). Requires more data and more complex evaluation, but produces fewer false positives for metrics with known diurnal or weekly patterns.

### Alert Evaluation Pipeline

Alert rules are evaluated periodically (every 15-60 seconds) by a rule evaluation engine. For each rule: execute the query against the time-series DB, evaluate the condition, track state (pending → firing → resolved). A pending state means the condition is true but hasn't held for the required duration. This state machine prevents alerts from firing and resolving on every evaluation tick during intermittent conditions.

## Alert Routing and Deduplication

Multiple firing alerts should not generate duplicate pages. An alert manager (Prometheus Alertmanager, PagerDuty) deduplicates alerts by grouping them by labels (service, environment, severity). Grouping: related alerts (all pods of the same service failing) are bundled into one notification. Inhibition: a high-severity alert (service down) suppresses lower-severity alerts from the same service (high latency — obviously, since it's down). Silences: suppress alerts during known maintenance windows.

## Notification Channels

Route by severity: P1 (customer-impacting) → PagerDuty (wakes on-call), Slack #incidents; P2 (degraded, not down) → Slack #alerts, email; P3 (warning, no action needed) → Slack #monitoring only. On-call rotation integration: alerts route to the current on-call engineer based on a rotation schedule. Escalation: if the primary on-call doesn't acknowledge within N minutes, escalate to secondary.

## Monitoring the Monitoring System

The monitoring system must itself be monitored. Dead man's switch: the monitoring system continuously sends a heartbeat to an external watchdog service. If the heartbeat stops, the watchdog pages the team — detecting monitoring outages. Monitor scrape success rates: if a target stops being scraped, alert. Use multiple independent monitoring systems for critical infrastructure (primary Prometheus + secondary DataDog) to avoid single points of failure in the observability stack.

## SLO-Based Alerting

Rather than alerting on raw metrics, alert on error budget burn rate. If a 30-day SLO allows 0.1% error rate, the monthly error budget is 43 minutes of downtime. Alert when the burn rate exceeds 14.4x normal (consuming the entire budget in 2 hours). This approach (from Google's SRE book) focuses alerts on SLO risk rather than individual metrics, reducing alert fatigue from metrics that exceed thresholds without actually threatening the SLO.

See also: [Meta Interview Guide 2026: Facebook, Instagram, WhatsApp Engineering](https://www.techinterview.org/post/3233460272/meta-interview-guide-2026-facebook-instagram-whatsapp-engineering/)

See also: [Uber Interview Guide 2026: Dispatch Systems, Geospatial Algorithms, and Marketplace Engineering](https://www.techinterview.org/post/3233460273/uber-interview-guide-2026-dispatch-systems-geospatial-algorithms-and-marketplace-engineering/)

See also: [Netflix Interview Guide 2026: Streaming Architecture, Recommendation Systems, and Engineering Excellence](https://www.techinterview.org/post/3233460274/netflix-interview-guide-2026-streaming-architecture-recommendation-systems-and-engineering-excellence/)

See also: [Scale AI Interview Guide 2026: Data Infrastructure, RLHF Pipelines, and ML Engineering](https://www.techinterview.org/post/3233460275/scale-ai-interview-guide-2026-data-infrastructure-rlhf-pipelines-and-ml-engineering/)

See also: [LinkedIn Interview Guide 2026: Social Graph Engineering, Feed Ranking, and Professional Network Scale](https://www.techinterview.org/post/3233460278/linkedin-interview-guide-2026-social-graph-engineering-feed-ranking-and-professional-network-scale/)

See also: [Airbnb Interview Guide 2026: Search Systems, Trust and Safety, and Full-Stack Engineering](https://www.techinterview.org/post/3233460279/airbnb-interview-guide-2026-search-systems-trust-and-safety-and-full-stack-engineering/)

See also: [Databricks Interview Guide 2026: Spark Internals, Delta Lake, and Lakehouse Architecture](https://www.techinterview.org/post/3233460280/databricks-interview-guide-2026-spark-internals-delta-lake-and-lakehouse-architecture/)

See also: [Anthropic Interview Guide 2026: Process, Questions, and AI Safety](https://www.techinterview.org/?p=3233460282)

See also: [Atlassian Interview Guide](https://www.techinterview.org/post/3233459919/atlassian-interview/)

See also: [Coinbase Interview Guide](https://www.techinterview.org/post/3233459920/coinbase-interview/)

See also: [Shopify Interview Guide](https://www.techinterview.org/post/3233459924/shopify-interview/)

See also: [Snap Interview Guide](https://www.techinterview.org/post/3233459923/snap-interview/)

See also: [Lyft Interview Guide 2026: Rideshare Engineering, Real-Time Dispatch, and Safety Systems](https://www.techinterview.org/post/3233460306/lyft-interview-guide-2026-rideshare-engineering-real-time-dispatch-and-safety-systems/)

See also: [Stripe Interview Guide 2026: Process, Bug Bash Round, and Payment Systems](https://www.techinterview.org/post/3233460268/stripe-interview-guide-2026-process-bug-bash-round-and-payment-systems/)