The web crawler question looks trivial: fetch a page, pull out its links, fetch those, repeat. Interviewers keep asking it because that naive loop falls apart the second you point it at the actual web. You will re-fetch the same URL a million times, get your IP blocked by an angry sysadmin, wander into a calendar that generates pages forever, and run out of memory tracking what you have already seen. The design is about controlling all of that.
A strong answer treats the crawler as a pipeline with a few hard problems buried inside it: not re-visiting URLs, not overloading any single site, and surviving the fact that fetching a billion pages takes days and machines will die partway through. Get the URL frontier right and most of the rest follows.
Pin down the scope before you draw a box
The first mistake is designing for “the whole internet” when the interviewer wanted something narrower. Ask what the crawler feeds. A search index has different needs than a price-monitoring bot or a training-data collector for a language model. Settle a few things out loud: are we crawling the open web or a fixed set of domains, HTML only or also PDFs and images, a one-time snapshot or a continuous recrawl that keeps content fresh, and do we need to render JavaScript with a headless browser or is raw HTML enough. Rendering changes the cost by an order of magnitude, so name it early.
For the rest of this walkthrough, assume the common version: crawl the open web, HTML only, continuously, targeting one billion pages a month, storing raw HTML for a separate indexing service to consume.
The numbers that shape the design
Back-of-envelope math tells you which parts have to be distributed and which fit on one box. A billion pages a month is about 385 pages per second on average, and you plan for roughly double that at peak.
| Quantity | Estimate | Where it comes from |
|---|---|---|
| Target crawl rate | ~385 pages/sec average, ~770 at peak | 1 billion pages over 30 days |
| Average HTML page size | ~500 KB raw, ~100 KB compressed | typical modern page payload |
| Download bandwidth | ~1.5 Gbps average, ~3 Gbps peak | 385/sec at 500 KB each |
| Raw HTML stored per month | ~100 TB compressed | 1 billion pages at 100 KB each |
| URLs tracked as “seen” | ~10 billion (you discover far more than you fetch) | ~10 links per page, heavily overlapping |
| Bloom filter for seen URLs | ~12 GB at a ~1% false-positive rate | 10 billion URLs at ~10 bits each |
The last row is the one that surprises people. Ten billion URLs in a plain hash set would need something like a terabyte of RAM and a disk lookup on every link. A Bloom filter answers “have I seen this?” in a handful of bit checks and fits in memory. The catch is that a Bloom filter can report a URL as seen when it was not, so you occasionally skip a page you meant to crawl. For a web crawler that is a fine trade, because the web is redundant and you will usually reach that page through another link.
The pipeline, end to end
Start with seed URLs and push them into the URL frontier, the scheduler that decides what to fetch next and when. A pool of fetcher workers pulls URLs from the frontier, resolves the hostname through a caching DNS layer, downloads the page, and hands the HTML to a parser. The parser writes the raw content to blob storage, extracts outbound links, and passes them through two filters before they can re-enter the frontier: a content check that asks whether we already stored this exact page, and a URL check that asks whether we already queued this address. Whatever survives both filters gets normalized and fed back into the frontier. That cycle is the whole crawler. Every hard part is one of those boxes doing more than it looks like.
Why the URL frontier is the actual design
A single FIFO queue fails immediately, because it interleaves URLs from every host at random and you end up firing fifty simultaneous requests at one small server. The frontier has to do two jobs at once: crawl important pages sooner, and never hit one host too hard. Those goals pull in different directions, so the standard design splits the frontier into two layers of queues.
The front queues handle priority. A prioritizer scores each URL by signals like PageRank, freshness, or how often the page changes, then drops it into one of several front queues ranked high to low. The back queues handle politeness. A router guarantees that every back queue holds URLs for exactly one host, and each fetcher worker binds to one back queue at a time. A small heap tracks the earliest moment each host is allowed to be contacted again. A worker takes the host whose timer has expired, fetches one page, then pushes the host back with a new timestamp derived from that site’s crawl delay. That structure is what keeps a thousand parallel workers from turning into an accidental denial-of-service attack against one blog.
Politeness also means reading robots.txt before you touch a host and caching it, honoring any Crawl-delay it specifies, and defaulting to a conservative gap of a second or two per host when it says nothing. Skipping this is how crawlers get IP-banned and how companies get nastygrams from site owners.
Two kinds of duplicate, two different tools
Deduplication runs at two levels, and candidates often collapse them into one. URL deduplication stops you from queuing the same address twice. Before you check, normalize: lowercase the host, drop the fragment after #, sort query parameters, resolve relative paths, and strip known tracking parameters. Example.com/a, example.com/a/, and example.com/a#top are one page. After normalizing, the Bloom-filter-plus-store check decides whether it is new.
Content deduplication is separate and catches a different problem: different URLs serving the same bytes. Mirror sites, session IDs baked into URLs, and printer-friendly versions all produce identical or near-identical content. A checksum such as MD5 or SHA-1 over the page body catches exact copies. Near-duplicates, where a page differs only in a timestamp or an ad slot, need a similarity hash like SimHash so you can group pages within a small Hamming distance and keep one. Without this, a single templated site floods your storage with millions of pages that say nothing new.
DNS and the traps that quietly kill throughput
DNS resolution is the bottleneck nobody mentions until it bites. Every fetch needs an IP, resolvers can take tens to hundreds of milliseconds, and a synchronous lookup per page caps your crawl rate long before bandwidth does. Run an asynchronous resolver with an aggressive local cache keyed by host, and you get most of that time back.
Then there are crawler traps, structures that generate URLs without end. An event calendar with a “next month” link runs to the year 3000. Faceted search on a shopping site multiplies filters into billions of combinations. Guard against them with a maximum crawl depth per seed, a cap on pages per domain, a limit on URL length, and pattern detection for query parameters that only permute. These caps feel arbitrary until one misconfigured site eats your whole frontier.
Making it distributed without breaking politeness
One machine cannot hold a 12 GB Bloom filter, a multi-terabyte frontier, and saturate several gigabits of bandwidth, so you shard. Partition by hash(host) and assign each host to exactly one crawler node. That single decision buys you two things: politeness stays correct because all requests to a host flow through one node that can pace them, and dedup stays local because a host’s URLs always land on the same machine. Use consistent hashing so adding or losing a node reshuffles only a slice of hosts rather than the entire map.
Because a full crawl runs for days, treat worker death as normal rather than exceptional. Persist the frontier to durable storage, a partitioned log like Kafka or disk-backed queues, so a crashed worker’s pending URLs survive. Checkpoint progress and make fetches idempotent, so re-crawling a page you already stored wastes a request but corrupts nothing.
Freshness, and what the follow-ups probe
A continuous crawler has to decide when to come back. Re-fetching every page on a fixed schedule spends most of your budget on pages that never change. The better approach estimates each page’s change rate from its history and recrawls high-churn pages like a news homepage far more often than static ones like a decade-old blog post. This is where the interviewer sees whether you think of the crawl as a fixed job or a living system.
The follow-up questions carry most of the signal, and they tend to sound like:
- “Your Bloom filter says a URL is seen, but you are not certain. What do you do?”
- “Two workers discover the same new host at the same instant. Which one crawls it?”
- “A site returns 200 for every URL, including ones that do not exist. How does your crawler notice?”
- “How would you politely crawl a site that has no robots.txt and a slow server?”
None of those have one right answer. They check whether you understand that a crawler spends most of its life deciding which page to touch next rather than downloading pages, and doing it without being a jerk to the servers on the other end. Name the frontier structure, split the two kinds of deduplication, be specific about politeness, and say plainly where the trade-offs bite, and you have shown the thing they are grading.
Keep sharpening your system design:
