What is a URL frontier in a web crawler?

A URL frontier is a priority queue of URLs waiting to be crawled. It prioritizes URLs based on factors like estimated PageRank and content freshness to ensure high-value pages are crawled first.

How does a web crawler respect robots.txt?

The crawler fetches and parses robots.txt for each domain before crawling any pages. It caches the result in Redis with a 24-hour TTL per domain to avoid repeated fetches, and respects the Crawl-delay directive with a default of 1 second between requests.

How does content deduplication work in a distributed crawler?

Duplicate URLs are filtered using a bloom filter on seen URLs before enqueuing. Duplicate content is detected using SimHash of the page body, which enables near-duplicate detection even when pages differ slightly.

How do you schedule re-crawls in a web crawler?

Re-crawl frequency is based on content change rate. Static pages are re-crawled every 7 days, while frequently updated content like news pages is re-crawled every hour. The crawler tracks change frequency over time to tune intervals dynamically.

Low Level Design: Web Crawler

URL Frontier

Priority queue of URLs to crawl, prioritized by PageRank estimate + freshness. Interviewers probe how you stop one busy domain from starving the queue: split the frontier into a priority tier that decides what to crawl next and a set of per-domain FIFO queues that decide when, so ordering and politeness stay separate concerns. Be ready to explain how a new URL’s priority is scored and how you cap queue growth when the crawl discovers links faster than workers can drain them.

CrawlJob Table

CrawlJob (
  id,
  url,
  domain,
  priority,
  status: pending/crawling/done/failed,
  last_crawled_at,
  next_crawl_at,
  http_status,
  content_hash
)

Crawl Flow

Dequeue URL → check robots.txt (cached per domain) → fetch URL → parse HTML → extract links → deduplicate links via seen-URL bloom filter → enqueue new URLs → store content.

Politeness

Per-domain crawl delay (respect Crawl-delay in robots.txt, default 1s) enforced via a per-domain rate limiter. The detail interviewers look for is that the delay is enforced per host, not globally — a token bucket keyed on domain lets fast domains stay fast while slow ones are throttled, and it survives the crawl fanning out across many workers.

robots.txt cache: Redis, 24h TTL per domain.

Content Deduplication

SimHash of page content → near-duplicate detection. SimHash maps each page to a fixed-length fingerprint where similar pages land within a small Hamming distance, so you catch near-duplicates — boilerplate templates, syndicated articles, URLs with tracking params — that an exact content hash would treat as brand new. A common rule is to flag pages within a Hamming distance of 3 bits over a 64-bit fingerprint; expect a follow-up on why you compare fingerprints instead of the raw text.

Distributed Architecture

Multiple crawler workers coordinate via shared URL frontier in Redis; each worker claims URLs by domain to respect politeness. Shard work by hashing the domain so every URL for a host lands on the same worker, which keeps rate limiting and Crawl-delay correct without cross-worker coordination. The usual follow-up is failure handling: put a lease or visibility timeout on claimed URLs so a crashed worker’s in-flight jobs are re-queued instead of silently lost.

Re-crawl Scheduling

Static pages every 7d, frequently-updated pages (news) every 1h based on change frequency detection. Drive the interval from how often content_hash actually changes between crawls rather than a fixed table: back off pages that never change and tighten intervals for pages that update often. A concrete adaptive scheme is to double the interval after an unchanged crawl and halve it after a change, bounded by a sensible min and max.

DNS Cache

Cache DNS lookups per domain (5min TTL) to avoid repeated lookups. Resolving the same host on every fetch adds latency and hammers your resolver, so cache resolutions and honor the record’s own TTL where you can. Worth mentioning: pre-warm DNS for high-priority domains and bound the cache with LRU eviction so memory stays flat as the crawl grows.