The whole course ▾

System Design · Lesson 32 of 39 · Week 7

Design a web crawler

Updated 22 Oct 2026 · By Akshay, software engineer, India · 5 min read

Crawl a billion web pages a month, without overloading a single website on the way. Today: design a web crawler.

The video for this lesson is coming soon.

New System Design videos go up at 6 pm. Subscribe on YouTube so you don't miss it.

Got a doubt? Ask in the comments under the video.Subscribe

Reading the web

Reading the web
What it must do
Start from seed URLs, follow
links
Download and store each page
Never overload a website
Revisit pages that change
The numbers
1B pages a month ÷ 30 ~ 33M a
day
33M ÷ 100,000 s ~ 400 pages a
second
× ~100 KB a page ~ 3 TB a day
Each fetch takes ~1 s:
thousands in flight
WHAT THE NUMBERS DECIDE
Downloadingiseasy.
Whattofetchnext,
politely,andnevertwice:
that'sthedesign.

Every search engine, and every AI model trained on the web, starts with a crawler: a system that downloads pages, finds their links, and follows them. It sounds like a loop of five lines. At scale, it's one of the classic design questions.

The requirements. Start from a list of seed URLs, like the front pages of big sites, and follow every link found. Download each page and store it, for the search index to read later. Never overload a website: a crawler that knocks a small business offline is a crawler that gets blocked. And revisit pages that change, so the copy stays fresh.

A billion pages a month is about thirty-three million a day: around four hundred pages a second. At roughly a hundred kilobytes a page, that's three terabytes of new pages a day. And since a single fetch can take a second or more, waiting on slow websites, we need thousands of fetches in flight at once.

So say it plainly: downloading pages is easy. The design is everything around it: deciding what to fetch next, being polite to every website, and never fetching or storing the same thing twice. Those three questions give us the rest of this lesson.

The crawl loop

The crawl loop
URL
frontier
what to
fetch
next
Fetchers
DNS
cache
Websites
Page
storage
Link
extractor
next URL
1
resolve
2
GET page
3
save page
4
new page
5
new links
6
THE FRONTIER IS THE BRAIN
Thefrontierdecides
order:importantpages
first,andnowebsitehit
toooften.

Here's the loop, drawn as a system.

The URL frontier holds every URL waiting to be fetched. Fetchers take the next one, look up the site's address through a DNS cache, because lesson one's DNS lookups would otherwise add time to every fetch, and download the page from the websites out there. They save it to page storage, object storage again. A link extractor reads each new page, finds its links, and adds them back to the frontier. Round and round, a billion times a month.

Notice where the intelligence lives: in the frontier. It's not a simple queue. It decides which pages matter most, like a news site's front page before an old forum post, and it decides when each website may be visited again. The fetchers just do what they're told.

Being polite

Being polite
Politeness
1
One queue per website
2
One request at a time per site
3
A pause between requests
4
Obey robots.txt
ROBOTS.TXT
Asite'srobots.txtsays
whichpathscrawlersmay
visit,andsometimeshow
often.

Four hundred pages a second is nothing for Wikipedia, and a disaster for a small bakery's website. Politeness is built into the frontier.

The frontier keeps one queue per website. Each website's queue is handled by one fetcher at a time, so a site gets one request at a time, never fifty. Between requests to the same site, there's a pause, a second or more, longer for small or slow sites. And before crawling a site at all, the crawler reads its robots dot txt.

Robots dot txt is a small file at the root of every site, which tells crawlers which paths they may and may not visit, and some sites add how long to wait between requests. A good crawler fetches it first, caches it for a while, and obeys it. It's not enforced by anything except good manners, and the threat of being blocked.

Never twice

Never twice
Seen it already?
1
Normalise URLs first
2
A Bloom filter of URLs seen
3
A hash of each page's content
4
Fingerprints for near-duplicates
THE BLOOM FILTER
Answers"definitelynot
seen"or"probablyseen",
usingabout10bitsper
URL.

The web is full of repetition: the same page under many URLs, and the same content on many pages. Fetching it all again would waste most of our effort.

First, normalise URLs: lowercase the host, drop tracking parameters, sort the rest, so one page doesn't look like ten. Then check a Bloom filter of URLs already seen. After fetching, hash the page's content: if two URLs return identical pages, store one. And for pages that are almost identical, like the same article with a different date at the top, compare similarity fingerprints, such as SimHash.

A Bloom filter is a compact bit array that answers one question: have I seen this before? It can say definitely not, or probably yes. With about ten bits per URL, it's wrong only around one time in a hundred, so a billion URLs fit in about a gigabyte of memory. The cost of a mistake is small: once in a while, we skip a page we hadn't actually seen.

Scale, traps and freshness

Scale, traps and freshness
Running at scale
->
Shard the frontier by website
->
Thousands of async fetches per machine
->
Crawler traps: limit depth per site
When to crawl again
->
News front pages: every few minutes
->
Most pages: days or weeks
->
Learn from how often it changed

Three more problems turn this from a weekend script into a real crawler.

Shard the frontier by a hash of the website's name, consistent hashing from lesson thirteen, so each website lives on exactly one machine, and politeness stays a local decision. Each machine runs thousands of async fetches at once, because fetching is mostly waiting. And watch for crawler traps: a calendar with a next month link that never ends, or URLs that grow forever. Limit the depth and the number of pages per site.

And freshness. News front pages change all the time, so revisit them every few minutes. Most pages change rarely, so every few days or weeks is enough. The crawler learns each page's rhythm: if it changed on the last three visits, come back sooner; if it hasn't changed in months, come back later. Sitemaps, which sites publish to list their pages, give useful hints too.

In the interview

In the interview
SAY THIS IN THE INTERVIEW
Aprioritisedfrontierwith
onequeueperhost,sharded
byhost;Bloomfiltersand
contenthashestodedupe.
Follow-ups to expect
->
How do you stay polite?
->
How do you avoid duplicates?
->
A machine dies mid-crawl?
THE COMMON MISTAKE
OnebigFIFOqueue:a
singlebusywebsitegets
hammered,andimportant
pageswaitbehindjunk.

The web crawler is a classic, because it's all about queues, deduplication, and doing the right thing at scale.

Say it like this: a URL frontier that prioritises important pages and keeps one queue per host, sharded across machines by host, so each site is crawled politely, obeying robots dot txt. URLs are deduplicated with a Bloom filter, and pages with content hashes. And recrawl frequency is learned from how often each page changes.

Then expect these. How do you stay polite? One queue per host, one request at a time, a delay, and robots dot txt. How do you avoid duplicates? Normalised URLs, a Bloom filter, and content hashes. And a machine dies mid-crawl? Its frontier shard is saved to disk regularly, so another machine picks up its websites and carries on.

The common mistake: one big first-in, first-out queue of URLs. A popular site's thousands of links come out together, and get fetched all at once, while important pages wait behind endless junk. The frontier must manage both order and politeness.

Quick check

Quick check
?
Why does the frontier keep
one queue per website?
Politeness: each site gets one
request at a time, with a pause
between them.
?
The Bloom filter says "probably
seen" for a URL it has never
seen. What's the cost?
One page skipped this time: a
small, acceptable loss.
Remember
The frontier decides order
and politeness
One queue per host, sharded
by host
Bloom filters and hashes:
never twice
Quick check

Why does the frontier keep one queue per website?

Show the answer

Politeness: each site gets one request at a time, with a pause between them.

Quick check

The Bloom filter says "probably seen" for a URL it has never seen. What's the cost?

Show the answer

One page skipped this time: a small, acceptable loss.

Remember: the frontier decides what's fetched next, and when each site may be visited. One queue per host, and the frontier sharded by host. And Bloom filters plus content hashes, so nothing is fetched or stored twice. Tomorrow: suggestions that appear as you type.

Practise with flashcardsYour progress stays on this device.
New System Design videos at 6 pm

Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.