web crawler – techinterview

techinterview.org

Companies
Cheat Sheets
Resume

Topics

System Design
Algorithms
Coding Patterns
AI / ML
Quant Finance
Frontend
Browse all topics

Companies

All companies
AI-native company guides
AI interview difficulty index
Company guides category

Roadmaps

Beginner (0–1y)
Intermediate (2–4y)
Advanced (5+y)

Tools

All tools
Interview Prompt Library
Video Interview Tech
Bootcamp Directory
Study plan generator
Resume / ATS checker
Total comp calculator
Big-O visualizer
STAR answer builder

Cheat Sheets

Big-O cheat sheet
Algorithm patterns
Problems by difficulty

More

Resume
About
Contact

Skip to content

techinterview

Home
All Problems
By Topic
By Difficulty
Company Guides
System Design
Algorithms
AI/ML Questions
Resume Guides
Quant Finance
Interview Prep

techinterview

About
Advanced Interview Preparation Guide (5+ Years)
AI Startup & AI Lab Interview Guides
AI-Startup Interview Difficulty Index (2026)
Beginner Interview Preparation Guide (0-1 Years)
Big-O Cheat Sheet: Time and Space Complexity Guide
Big-O Complexity Visualizer
Blog
Bootcamp Directory
Breaking Into Wall Street: Tech Interview Guides (2026)
Browse All Topics
Coding Interview Patterns: The Complete Catalog
Coding Interview Study Plan Generator
Common Algorithm Patterns Cheat Sheet
Comp Explorer: Compensation by Company and Level (2026)
Complete Problem Index
Contact
Free Interview Prep Tools
Free Resume & ATS Checker
Geography of the Interview Loop (2026)
Home
Intermediate Interview Preparation Guide (2-4 Years)
Interview Cheat Sheet Library
Interview Format Heatmap: Round Types by Company Segment (2026)
Interview Prep Prompts
Interviewing in the AI Era (2026)
Media Kit
Privacy Policy
Problems by Difficulty
Problems by Topic
Quant & HFT Firm Interview Difficulty Index (2026)
Quant & HFT Firm Interview Guides
STAR Method Answer Builder
System Design Interview Guides
Tech Total Comp & Offer Calculator
Terms of Service
Video Interview Tech

Advertisements

How to design a web crawler that stays polite at a billion pages

techinterview.org / July 29, 2026

7 min read A walkthrough of the web crawler design question, from the URL frontier and politeness to deduplication and the follow-ups that trip candidates up. Read article

newsletter

What's actually being asked right now

Interview patterns & comp trends, straight to your inbox.

Leave this field empty

No spam. Unsubscribe anytime.

worth a read

Mars For The Rest of Us — a weekly-or-more deep dive on the technical side of Mars exploration: rocket propulsion, microbiology, mission architecture, and everything in between. Written by Maciej Ceglowski.

Read it on Substack →

Advertisements

techinterview.org // engineers writing for engineers

Topics

System Design
Coding Patterns
Company Guides
Cheat Sheets
Problem Index

Study Paths

Beginner (0-1y)
Intermediate (2-4y)
Advanced (5+y)

Resources

Big-O Cheat Sheet
Algorithm Patterns
Problems by Difficulty
Problems by Topic

Site

About
Contact
Privacy Policy
Terms of Service

© 2026 techinterview.org — all rights reserved RSS Contact

About
|
Contact
|
Privacy Policy
|
Terms of Service

© 2026 Tech Interview Dot Org. All rights reserved.

Scroll to Top

// feedback

What's missing, wrong, or worth adding? One line helps.

Sent — thank you. We read every one.

This site uses cookies for analytics and to display ads via Google AdSense. By continuing to use the site you consent to our use of cookies. See our Privacy Policy for details and opt-out links.