Why use HTTP HEAD instead of GET for link checking?

HEAD requests return only response headers without the response body, saving significant bandwidth when checking large numbers of links. The status code and redirect behavior are identical to GET, making HEAD sufficient for determining whether a link is broken.

How do you detect redirect loops in a link checker?

Maintain a seen-URL set per redirect chain. If a redirect destination is already in the set, a loop is detected and the link is marked with error_type=redirect_loop. Following is also capped at a maximum of 10 redirects to handle excessive chains.

How should a link checker handle external domains?

External links are checked with HTTP HEAD requests to verify they resolve, but the checker does not crawl or follow links on external domains. An ignore list can be configured to skip known slow but valid third-party links to reduce false positives.

What retry and scheduling strategy works best for link checking?

Retry each failing link 3 times with a 1-second delay before marking it as broken to filter transient errors. Run a full site check weekly. Use a watch mode that re-checks previously identified broken links daily so issues are tracked and reported promptly.

Low Level Design: Link Checker Service

Data Model

CheckJob Table

CheckJob (
  id,
  target_domain,
  status: queued/running/completed,
  started_at,
  completed_at,
  total_links,
  broken_count
)

LinkResult Table

LinkResult (
  id,
  job_id,
  source_url,
  target_url,
  http_status,
  redirect_chain JSONB,
  latency_ms,
  error_type: timeout/dns/ssl/404/500/redirect_loop,
  checked_at
)

Store broken_count and total_links as denormalized counters on CheckJob so the summary view does not scan every row. Reports read LinkResult by job_id and filter on error_type, so indexing those two columns is what keeps the report query fast as result counts grow into the millions. Interviewers often probe why the redirect chain is a single JSONB column rather than its own table — the answer is that you never query inside it, you only display it.

Crawl Flow

Start from seed URL → extract all anchor href and src attributes → resolve relative URLs to absolute → queue for HTTP HEAD request (prefer HEAD over GET to save bandwidth).

Keep a per-job seen set so the same URL is checked once even when linked from dozens of pages. A common follow-up is what to do when HEAD is not supported: some servers return 405 or 501 for HEAD, so fall back to a ranged GET (Range: bytes=0-0) before deciding the link is broken.

Redirect Handling

Follow up to 10 redirects → detect loops via seen-URL set.

Record the full redirect_chain and store the final destination’s status separately from the intermediate 301/302 codes, since a link that 301s to a 404 is broken even though the first hop succeeded. Distinguish a genuine loop (a URL that reappears in the seen set) from a chain that is merely long but terminates — only the former gets error_type=redirect_loop.

External Link Handling

Check external links but do not crawl external domains.

Verify that the external URL resolves with a HEAD request, but never parse its HTML or follow its links — that would turn a site checker into an unbounded web crawler. Treat 403 and 429 responses from bot protection as ambiguous rather than broken, because many large sites block automated clients; flagging those as failures is the main source of false positives.

Concurrency

N workers per job, per-domain rate limit (max 5 rps).

Use a bounded worker pool per job and key the rate limiter by target domain so one slow host throttles only its own requests instead of stalling the whole crawl. A token-bucket limiter per domain is the usual answer here; interviewers probe whether your 5 rps cap is global or per-domain, because a global cap wastes capacity when a job spans many hosts.

Retry Policy

Retry 3x with 1s delay before marking as broken.

Only retry failures that could be transient — timeouts, DNS hiccups, and 5xx responses — and do not retry a clean 404, which is deterministic and wastes requests. Exponential backoff with jitter is stronger than a fixed 1s delay because it avoids hammering a host that is already struggling, and telling transient from permanent errors is a point interviewers push on.

Report Generation

Group by error_type + HTTP status, sorted by frequency.

Dedupe by target_url so a link that is broken on 50 pages appears once, with the list of source_url values that reference it, so an author can find every anchor to fix. A frequent extension is trend reporting — comparing this run’s broken set against the previous run to show what newly broke versus what was fixed.

Scheduling

Run full check weekly; watch mode re-checks known-broken links daily.

The full weekly crawl is expensive, so watch mode narrows the daily run to the small known-broken set to catch fixes quickly without re-scanning the whole site. Guard against overlapping runs of the same job with a lock or single-flight check, since a slow crawl that outlasts its schedule would otherwise stack a second copy on top of itself.

Notifications

Email report with broken link count + CSV export.

Notify only when broken_count changes or crosses a threshold rather than on every run, or the report becomes noise people stop reading. The CSV should carry source_url, target_url, http_status, and error_type per row so authors can sort and filter it directly, and a webhook or Slack option is a natural addition for teams that do not want email.

Ignore List

Exclude known third-party links that are slow but not broken.

Match entries by URL pattern or domain, and separate a true ignore (never check) from an allow-slow rule (check but with a longer timeout). This is the lever for false positives — sites behind bot protection that return codes like 999 or a hard 403 to automated clients belong here so they stop showing up as broken every run.