Low Level Design: Request Routing Service

What Is a Request Routing Service?

A request routing service directs incoming HTTP requests to the appropriate upstream based on rules such as path, headers, and traffic weights. It enables canary deployments, A/B testing, and blue-green promotions without application changes.

Route Types

Exact match — /api/v1/users matches only that path
Prefix match — /api/v1/ matches any path starting with that prefix
Regex match — /api/v[0-9]+/users for version-agnostic routing

Priority order: exact > prefix > regex. Within the same type, higher priority value wins.

Routing Rule Table

routing_rules (
  id             BIGSERIAL PRIMARY KEY,
  match_type     TEXT NOT NULL,         -- exact, prefix, regex
  pattern        TEXT NOT NULL,
  method         TEXT NOT NULL,         -- GET, POST, *, etc.
  headers_match  JSONB,                 -- e.g. {X-Canary: true}
  weight         INT DEFAULT 100,       -- 0-100, used for weighted split
  upstream       TEXT NOT NULL,
  priority       INT DEFAULT 0
)

Header-Based Routing

Match on arbitrary request headers to direct traffic to a specific upstream. Example: requests with X-Canary: true route to the canary upstream; all others route to stable. Useful for internal testing and gradual feature rollout.

Rule: headers_match = {X-Canary: true} -> upstream = canary
Default rule: upstream = stable

Weight-Based Traffic Splitting

Assign integer weights to competing rules for the same pattern. Use weighted random selection at runtime:

canary rule:  weight=5  → 5% of traffic
stable rule:  weight=95 → 95% of traffic

Selection: rand(0,100) < 5 → canary, else stable

Weights are evaluated after header rules so explicit header overrides take precedence.

Sticky Routing

Use consistent hashing on session_id or user_id to ensure the same user always reaches the same upstream during a split. This prevents a user from seeing inconsistent behavior when they bounce between canary and stable across requests.

hash(user_id) % 100 < canary_weight → canary
else → stable

Traffic Mirroring

Shadow-copy requests to a test upstream asynchronously. The original request is proxied to the primary upstream and the response returned to the client immediately. The mirrored copy is sent to the shadow upstream in a fire-and-forget goroutine/thread. Shadow responses are discarded — useful for validating new service behavior under real traffic.

Route Evaluation Algorithm

On each request:

Check exact match rules — O(1) via hash map
Check prefix rules — use a trie for efficient longest-prefix match
Check regex rules — iterate sorted by priority, short-circuit on first match
Apply header conditions on matched rules
Apply weighted selection if multiple rules remain

Blue-Green Promotion

Phase 1: blue=100, green=0    (green deployed, no traffic)
Phase 2: blue=50,  green=50   (validation)
Phase 3: blue=0,   green=100  (full cutover)
Rollback: set blue=100, green=0 instantly

Because weight changes are propagated via Redis pub/sub (see below), rollback takes effect in under 1 second across all gateway instances.

Upstream Health Integration

The router only selects upstreams that are currently marked healthy by the health check service. If all weighted upstreams for a rule are unhealthy, return 503 Service Unavailable rather than routing to a known-bad instance.

Route Change Propagation

Admin API → DB write → Redis pub/sub publish
  → all gateway instances subscribe → in-memory reload (<1s)

No rolling restart required. Propagation latency is typically under 100ms on a local network.

Summary

A request routing service enables safe, gradual deployments through exact/prefix/regex matching, header overrides, weighted splits, and consistent hashing for stickiness. Hot-reloadable rules via Redis pub/sub make blue-green promotion and rollback near-instant.

{
“@context”: “https://schema.org”,
“@type”: “FAQPage”,
“mainEntity”: [
{
“@type”: “Question”,
“name”: “What is the difference between exact, prefix, and regex route matching?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Exact match applies only to a specific path like /api/v1/users. Prefix match applies to any path starting with a given prefix like /api/v1/. Regex match uses a pattern such as /api/v[0-9]+/users to handle version-agnostic routing. Priority order is exact first, then prefix, then regex — within the same type a numeric priority field breaks ties.”
}
},
{
“@type”: “Question”,
“name”: “How does weight-based traffic splitting work for canary deployments?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Each routing rule has an integer weight. For a given path, competing rules are selected using weighted random sampling. For example, a canary rule with weight 5 and a stable rule with weight 95 will send roughly 5% of traffic to canary and 95% to stable. Sticky routing via consistent hashing on user_id ensures the same user always reaches the same upstream during the split.”
}
},
{
“@type”: “Question”,
“name”: “How do you implement a blue-green deployment with a routing service?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Set the blue upstream weight to 100 and green to 0 while deploying the new version. Gradually shift weight — for example 50/50 for validation — then move to 0/100 for full cutover. Rollback is instant: set blue back to 100 and green to 0. Route weight changes propagate to all gateway instances via Redis pub/sub in under one second.”
}
},
{
“@type”: “Question”,
“name”: “What is traffic mirroring and when should you use it?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Traffic mirroring sends a copy of each request to a shadow upstream asynchronously while the real response comes from the primary upstream. The shadow response is discarded. It's used to validate a new service version under real production traffic without affecting users — useful before promoting a canary or testing a rewritten service.”
}
}
]
}