The whole course ▾
System Design · Week 7 revision
Week 7 revision: Case studies
Case study week in ten minutes: YouTube, Dropbox, a crawler, autocomplete and notifications, then six questions.
New System Design videos go up at 6 pm. Subscribe on YouTube so you don't miss it.
Got a doubt? Ask in the comments under the video.Subscribe
Lesson 30YouTube
This week, five famous systems, each built from the patterns of the weeks before. First, five hundred hours of video every minute.
Lesson thirty. The creator app uploads in resumable parts straight to object storage, and the original is kept forever. The video is cut into segments a few seconds long, and each one becomes a job on the transcode queue, so hundreds of transcode workers produce six qualities in parallel. The viewer's player fetches segments from a CDN edge, measuring its speed and switching quality between segments: adaptive streaming. Views are counted in batches, through a stream.
The interview line: resumable uploads to object storage, segment-parallel transcoding through queues and workers, adaptive bitrate streaming with HLS, every segment served from a CDN, and views counted asynchronously. And popular videos stay cached at every edge, while the long tail is served from fewer places.
Lesson 31Dropbox
Then a one-slide edit to a two-hundred-megabyte presentation, synced in seconds.
Lesson thirty-one, in three lines. Files are split into four-megabyte blocks, each named by the hash of its content, so a file is just a list of hashes. Only new blocks travel, and identical blocks are stored once, for everyone. Metadata lives in a consistent database, blocks in object storage, and every commit names the version it started from, so conflicts are caught and kept as conflicted copies.
The interview line: files as lists of content-hashed blocks in object storage, metadata and versions in a strongly consistent database, a notification channel so other devices fetch only what changed, and conflicted copies instead of guessed merges. And remember the insertion trap: cutting blocks by content keeps an early edit from shifting every block after it.
Lesson 32The web crawler
Then a billion pages a month, fetched without upsetting a single website.
Lesson thirty-two. The frontier is the brain: it fetches important pages first, and keeps one queue per host, sharded across machines by host. Politeness means one request at a time per site, a pause between them, and obeying robots dot txt. And nothing is fetched twice: normalised URLs, a Bloom filter of URLs seen, and content hashes for duplicate pages.
The interview line: a prioritised URL frontier with one queue per host, sharded by host, polite and obeying robots dot txt, deduplicating with a Bloom filter and content hashes, and recrawling each page as often as it changes. The crawl loop itself is simple; the frontier is where the design lives.
Lessons 33 and 34Autocomplete and notifications
And two systems that live on speed and on manners.
Lesson thirty-three. A trie, with the top ten completions stored at every node, so each keystroke is a single lookup. It's built offline from query logs, and served from memory. A trending layer, updated every few minutes, keeps up with the news, and a safety filter removes what should never be suggested. The browser caches popular prefixes too, and waits a moment between keystrokes.
Lesson thirty-four. Events go through preferences, quiet hours and templates, and are deduplicated. Then a queue per channel, push, text and email, so one slow provider stays contained, with retries and fallbacks. And separate lanes, so a fifty-million-user offer never delays a one-time password. Delivery receipts come back by webhook, like payments.
What's nextNext week: senior level
Next week is the last week of lessons, and the hardest: the topics that separate senior answers from good ones.
Google Docs, where many people type in the same document at once. A distributed job scheduler, cron for a whole company. Stream processing, counting billions of ad clicks exactly. Multi-region systems, for QuickBite's launch in a second country. And thinking like a senior engineer: the whole course, on one board.
Test yourself
What does a video player use to choose the quality of its next segment?
Show the answer
Its measured download speed for the last few segments.
You copy a 5 GB folder to a new place inside Dropbox. How much is uploaded?
Show the answer
Almost nothing: every block's hash is already stored.
Why shard the crawler's frontier by website?
Show the answer
Each site lives on one machine, so politeness is a local decision.
A calendar page links to "next month" forever. What protects the crawler?
Show the answer
Limits on depth and pages per site: it's a crawler trap.
Why build the autocomplete trie offline, instead of on each request?
Show the answer
Building is heavy; serving must be a fast lookup, so do the work once, ahead of time.
A sale goes out to 50 million users. How do one-time passwords stay fast?
Show the answer
Separate transactional queues: they never wait behind marketing.
Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.