The whole course ▾
System Design · Lesson 32 of 39 · Week 7
Design a web crawler
Crawl a billion web pages a month, without overloading a single website on the way. Today: design a web crawler.
New System Design videos go up at 6 pm. Subscribe on YouTube so you don't miss it.
Got a doubt? Ask in the comments under the video.Subscribe
Reading the web
Every search engine, and every AI model trained on the web, starts with a crawler: a system that downloads pages, finds their links, and follows them. It sounds like a loop of five lines. At scale, it's one of the classic design questions.
The requirements. Start from a list of seed URLs, like the front pages of big sites, and follow every link found. Download each page and store it, for the search index to read later. Never overload a website: a crawler that knocks a small business offline is a crawler that gets blocked. And revisit pages that change, so the copy stays fresh.
A billion pages a month is about thirty-three million a day: around four hundred pages a second. At roughly a hundred kilobytes a page, that's three terabytes of new pages a day. And since a single fetch can take a second or more, waiting on slow websites, we need thousands of fetches in flight at once.
So say it plainly: downloading pages is easy. The design is everything around it: deciding what to fetch next, being polite to every website, and never fetching or storing the same thing twice. Those three questions give us the rest of this lesson.
The crawl loop
Here's the loop, drawn as a system.
The URL frontier holds every URL waiting to be fetched. Fetchers take the next one, look up the site's address through a DNS cache, because lesson one's DNS lookups would otherwise add time to every fetch, and download the page from the websites out there. They save it to page storage, object storage again. A link extractor reads each new page, finds its links, and adds them back to the frontier. Round and round, a billion times a month.
Notice where the intelligence lives: in the frontier. It's not a simple queue. It decides which pages matter most, like a news site's front page before an old forum post, and it decides when each website may be visited again. The fetchers just do what they're told.
Being polite
Four hundred pages a second is nothing for Wikipedia, and a disaster for a small bakery's website. Politeness is built into the frontier.
The frontier keeps one queue per website. Each website's queue is handled by one fetcher at a time, so a site gets one request at a time, never fifty. Between requests to the same site, there's a pause, a second or more, longer for small or slow sites. And before crawling a site at all, the crawler reads its robots dot txt.
Robots dot txt is a small file at the root of every site, which tells crawlers which paths they may and may not visit, and some sites add how long to wait between requests. A good crawler fetches it first, caches it for a while, and obeys it. It's not enforced by anything except good manners, and the threat of being blocked.
Never twice
The web is full of repetition: the same page under many URLs, and the same content on many pages. Fetching it all again would waste most of our effort.
First, normalise URLs: lowercase the host, drop tracking parameters, sort the rest, so one page doesn't look like ten. Then check a Bloom filter of URLs already seen. After fetching, hash the page's content: if two URLs return identical pages, store one. And for pages that are almost identical, like the same article with a different date at the top, compare similarity fingerprints, such as SimHash.
A Bloom filter is a compact bit array that answers one question: have I seen this before? It can say definitely not, or probably yes. With about ten bits per URL, it's wrong only around one time in a hundred, so a billion URLs fit in about a gigabyte of memory. The cost of a mistake is small: once in a while, we skip a page we hadn't actually seen.
Scale, traps and freshness
Three more problems turn this from a weekend script into a real crawler.
Shard the frontier by a hash of the website's name, consistent hashing from lesson thirteen, so each website lives on exactly one machine, and politeness stays a local decision. Each machine runs thousands of async fetches at once, because fetching is mostly waiting. And watch for crawler traps: a calendar with a next month link that never ends, or URLs that grow forever. Limit the depth and the number of pages per site.
And freshness. News front pages change all the time, so revisit them every few minutes. Most pages change rarely, so every few days or weeks is enough. The crawler learns each page's rhythm: if it changed on the last three visits, come back sooner; if it hasn't changed in months, come back later. Sitemaps, which sites publish to list their pages, give useful hints too.
In the interview
The web crawler is a classic, because it's all about queues, deduplication, and doing the right thing at scale.
Say it like this: a URL frontier that prioritises important pages and keeps one queue per host, sharded across machines by host, so each site is crawled politely, obeying robots dot txt. URLs are deduplicated with a Bloom filter, and pages with content hashes. And recrawl frequency is learned from how often each page changes.
Then expect these. How do you stay polite? One queue per host, one request at a time, a delay, and robots dot txt. How do you avoid duplicates? Normalised URLs, a Bloom filter, and content hashes. And a machine dies mid-crawl? Its frontier shard is saved to disk regularly, so another machine picks up its websites and carries on.
The common mistake: one big first-in, first-out queue of URLs. A popular site's thousands of links come out together, and get fetched all at once, while important pages wait behind endless junk. The frontier must manage both order and politeness.
Quick check
Why does the frontier keep one queue per website?
Show the answer
Politeness: each site gets one request at a time, with a pause between them.
The Bloom filter says "probably seen" for a URL it has never seen. What's the cost?
Show the answer
One page skipped this time: a small, acceptable loss.
Remember: the frontier decides what's fetched next, and when each site may be visited. One queue per host, and the frontier sharded by host. And Bloom filters plus content hashes, so nothing is fetched or stored twice. Tomorrow: suggestions that appear as you type.
Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.