# Canary Analysis Service Low-Level Design: Metric Comparison, Statistical Tests, and Pass/Fail Verdict

Source: https://www.techinterview.org/post/3233470087/lld-canary-analysis/
Updated: 2026-04-17 · techinterview.org

## What Is a Canary Analysis Service?

A canary analysis service automates the statistical comparison of a new software version (the canary) against the stable baseline. Rather than relying on engineers to manually read dashboards during a canary deployment, the service continuously collects metrics from both populations, applies statistical tests, and emits a pass or fail verdict that can gate further rollout or trigger automatic rollback.

## Requirements

### Functional Requirements

- Accept a canary analysis configuration specifying metric sources, baseline and canary identifiers, and success criteria.

- Continuously fetch metric samples from a time-series backend (Prometheus, Datadog, or similar).

- Apply configurable statistical tests to compare canary and baseline distributions.

- Emit a verdict (PASS, FAIL, INCONCLUSIVE) after a configured observation window.

- Expose the verdict and per-metric results via API and webhook callback.

### Non-Functional Requirements

- Support concurrent analyses for dozens of simultaneous canary deployments.

- Complete verdict computation within 5 seconds of the observation window closing.

- Provide audit logs of every metric sample and intermediate verdict decision.

## Data Model

An **AnalysisRun** record is created when a deployment system initiates a canary check.

- **run_id** (UUID)

- **canary_version**, **baseline_version** (strings)

- **metric_configs** (JSONB array of metric name, query template, threshold, direction)

- **start_time**, **end_time**, **window_minutes**

- **status** (ENUM: running, pass, fail, inconclusive, cancelled)

Each metric comparison is stored as a **MetricResult** row: run_id, metric_name, canary_samples (float array), baseline_samples (float array), p_value, effect_size, verdict (pass/fail), evaluated_at.

## Core Algorithms

### Mann-Whitney U Test

The service uses the Mann-Whitney U (Wilcoxon rank-sum) test as its primary statistical test because it makes no assumption about the underlying distribution — latency and error-rate distributions are rarely normal. The test determines whether samples from the canary population are stochastically greater or less than the baseline population.

- Combine and rank all samples from both populations.

- Compute U statistics for both groups: U1 = R1 - n1*(n1+1)/2, where R1 is the sum of ranks for group 1.

- Convert U to a Z score for large samples (n > 20), then derive a two-tailed p-value.

- If p-value is below the configured significance level (default 0.05) and the effect is in the failing direction, mark the metric as failed.

### Effect Size Guard

Statistical significance alone is insufficient with large sample sizes. The service also computes a relative difference: (canary_median - baseline_median) / baseline_median. A metric only fails if both the p-value threshold is crossed AND the relative difference exceeds the configured tolerance (e.g. 5% regression in p99 latency).

## Scalability

Each AnalysisRun is handled by a dedicated worker goroutine that polls the metrics backend at a configurable interval (e.g. every 60 seconds). Workers are distributed across a pool of analysis nodes using a consistent hash on run_id, so a single run always lands on the same node for local state accumulation. A coordinator service tracks active runs and reassigns them if a worker node fails, using a heartbeat lease in Redis.

Metric queries are batched where the backend supports it. For Prometheus, a single range query per metric covers the full window, avoiding N+1 query patterns.

## API Design

- **POST /analyses** — start a new analysis run; returns run_id.

- **GET /analyses/{run_id}** — return current status, elapsed time, and per-metric intermediate results.

- **GET /analyses/{run_id}/results** — return the final verdict, per-metric verdicts, p-values, and sample summaries.

- **DELETE /analyses/{run_id}** — cancel an in-progress run.

Webhook delivery: when a run reaches a terminal status, the service POSTs a signed JSON payload to the registered callback URL. The payload includes run_id, verdict, failed_metrics array, and a summary of effect sizes. Delivery is retried up to 5 times with exponential backoff.

## Failure Modes

- **Metrics backend unavailable:** Samples collected so far are retained. The observation window is extended by the outage duration, up to a configured maximum extension.

- **Insufficient samples:** If either population has fewer than 10 data points at verdict time, the metric is marked INCONCLUSIVE rather than PASS or FAIL.

- **Worker crash:** The lease expires, the coordinator reassigns the run, and the new worker resumes from the last persisted sample set.

## Observability

Track active run count, per-run sample ingestion rate, metric query latency, and verdict distribution over time. Alarm when the inconclusive rate exceeds 20% across all runs — this typically indicates metrics backend instability or misconfigured queries.

See also: [Netflix Interview Guide 2026: Streaming Architecture, Recommendation Systems, and Engineering Excellence](https://www.techinterview.org/post/3233460274/netflix-interview-guide-2026-streaming-architecture-recommendation-systems-and-engineering-excellence/)

See also: [Scale AI Interview Guide 2026: Data Infrastructure, RLHF Pipelines, and ML Engineering](https://www.techinterview.org/post/3233460275/scale-ai-interview-guide-2026-data-infrastructure-rlhf-pipelines-and-ml-engineering/)

See also: [Anthropic Interview Guide 2026: Process, Questions, and AI Safety](https://www.techinterview.org/?p=3233460282)