# Low Level Design: Link Checker Service

Source: https://www.techinterview.org/post/3233471087/lld-link-checker/
Updated: 2026-07-26 · techinterview.org

## Data Model

### CheckJob Table


```
CheckJob (
  id,
  target_domain,
  status: queued/running/completed,
  started_at,
  completed_at,
  total_links,
  broken_count
)
```


### LinkResult Table


```
LinkResult (
  id,
  job_id,
  source_url,
  target_url,
  http_status,
  redirect_chain JSONB,
  latency_ms,
  error_type: timeout/dns/ssl/404/500/redirect_loop,
  checked_at
)
```


Store `broken_count` and `total_links` as denormalized counters on `CheckJob` so the summary view does not scan every row. Reports read `LinkResult` by `job_id` and filter on `error_type`, so [indexing](https://www.techinterview.org/post/3233461821/database-indexing-interview-guide/) those two columns is what keeps the report query fast as result counts grow into the millions. Interviewers often probe why the redirect chain is a single JSONB column rather than its own table — the answer is that you never query inside it, you only display it.

## Crawl Flow

Start from seed URL → extract all anchor `href` and `src` attributes → resolve relative URLs to absolute → queue for HTTP HEAD request (prefer HEAD over GET to save bandwidth).

Keep a per-job seen set so the same URL is checked once even when linked from dozens of pages. A common follow-up is what to do when HEAD is not supported: some servers return 405 or 501 for HEAD, so fall back to a ranged GET (`Range: bytes=0-0`) before deciding the link is broken.

## Redirect Handling

Follow up to 10 redirects → detect loops via seen-URL set.

Record the full `redirect_chain` and store the final destination's status separately from the intermediate 301/302 codes, since a link that 301s to a 404 is broken even though the first hop succeeded. Distinguish a genuine loop (a URL that reappears in the seen set) from a chain that is merely long but terminates — only the former gets `error_type=redirect_loop`.

## External Link Handling

Check external links but do not crawl external domains.

Verify that the external URL resolves with a HEAD request, but never parse its HTML or follow its links — that would turn a site checker into an unbounded web crawler. Treat 403 and 429 responses from bot protection as ambiguous rather than broken, because many large sites block automated clients; flagging those as failures is the main source of false positives.

## Concurrency

N workers per job, per-domain [rate limit](https://www.techinterview.org/post/3233474159/system-design-rate-limiter-token-bucket-sliding-window-leaky-bucket-distributed-rate-limiting-api-gateway/) (max 5 rps).

Use a bounded worker pool per job and key the rate limiter by target domain so one slow host throttles only its own requests instead of stalling the whole crawl. A token-bucket limiter per domain is the usual answer here; interviewers probe whether your 5 rps cap is global or per-domain, because a global cap wastes capacity when a job spans many hosts.

## Retry Policy

Retry 3x with 1s delay before marking as broken.

Only retry failures that could be transient — timeouts, DNS hiccups, and 5xx responses — and do not retry a clean 404, which is deterministic and wastes requests. Exponential backoff with jitter is stronger than a fixed 1s delay because it avoids hammering a host that is already struggling, and telling transient from permanent errors is a point interviewers push on.

## Report Generation

Group by `error_type` + HTTP status, sorted by frequency.

Dedupe by `target_url` so a link that is broken on 50 pages appears once, with the list of `source_url` values that reference it, so an author can find every anchor to fix. A frequent extension is trend reporting — comparing this run's broken set against the previous run to show what newly broke versus what was fixed.

## Scheduling

Run full check weekly; watch mode re-checks known-broken links daily.

The full weekly crawl is expensive, so watch mode narrows the daily run to the small known-broken set to catch fixes quickly without re-scanning the whole site. Guard against overlapping runs of the same job with a lock or single-flight check, since a slow crawl that outlasts its schedule would otherwise stack a second copy on top of itself.

## Notifications

Email report with broken link count + CSV export.

Notify only when `broken_count` changes or crosses a threshold rather than on every run, or the report becomes noise people stop reading. The CSV should carry `source_url`, `target_url`, `http_status`, and `error_type` per row so authors can sort and filter it directly, and a webhook or Slack option is a natural addition for teams that do not want email.

## Ignore List

Exclude known third-party links that are slow but not broken.

Match entries by URL pattern or domain, and separate a true ignore (never check) from an allow-slow rule (check but with a longer timeout). This is the lever for false positives — sites behind bot protection that return codes like 999 or a hard 403 to automated clients belong here so they stop showing up as broken every run.

See also: [Netflix Interview Guide 2026: Streaming Architecture, Recommendation Systems, and Engineering Excellence](https://www.techinterview.org/post/3233460274/netflix-interview-guide-2026-streaming-architecture-recommendation-systems-and-engineering-excellence/)

See also: [Atlassian Interview Guide](https://www.techinterview.org/post/3233459919/atlassian-interview/)

See also: [Scale AI Interview Guide 2026: Data Infrastructure, RLHF Pipelines, and ML Engineering](https://www.techinterview.org/post/3233460275/scale-ai-interview-guide-2026-data-infrastructure-rlhf-pipelines-and-ml-engineering/)