The whole course ▾
System Design · Lesson 36 of 39 · Week 8
Design a distributed job scheduler
QuickBite runs ten million scheduled jobs a day: payouts at midnight, reminders at noon, reports every hour. Today: a distributed job scheduler.
New System Design videos go up at 6 pm. Subscribe on YouTube so you don't miss it.
Got a doubt? Ask in the comments under the video.Subscribe
One cron machine
Every company has work that must happen at a certain time: pay the restaurants at midnight, remind customers about their saved basket at noon, build the sales report every hour. At first, one machine does it all.
That machine is the cron server: one machine, with one list of scheduled commands, called a crontab. At midnight it starts payouts. At noon, reminders. Every hour, reports. Simple, and for a small company, perfectly good. But QuickBite now has ten million scheduled jobs a day, from dozens of teams, and some of them move money.
Here's what goes wrong. At eleven fifty-nine, the cron server's disk fails, and it dies. Midnight passes, nobody notices, and two hundred thousand restaurants aren't paid. And even on a good day, one machine can't start ten million jobs, can't share them across many workers, and can't tell anyone whether a job ran, failed, or ran twice. We need cron, for a whole company.
Jobs and runs
First, the data model, because it splits the problem neatly in two.
A job is the definition: what to run, its schedule, like every day at midnight, how many retries it gets, and which team owns it. A run is one execution of that job at one moment: the midnight run of the twenty-ninth of November, its status, and who's working on it. Every time a run is created, the next one is worked out from the schedule and added too, so the runs table always holds what's due next.
Finding what's due is a simple query, made fast by lesson ten: index the runs by their start time. Every second, a scheduler asks: which pending runs are due by now? It gets back a small batch, and those are handed to workers. The scheduler itself does almost no work. And it can be several machines at once, because, as you'll see next, a run can only be claimed once.
Claiming a run
Many workers, many due runs. Each run must go to exactly one worker, and must not be lost if that worker dies.
Two workers both see the midnight payout run as due, in the runs table. Each tries to claim it with a conditional write, from lesson twenty-eight: set status to running, worker to me, and a lease until five minutes from now, where the status is still pending. Exactly one write succeeds: worker seven gets it, and worker nine's write changes nothing. While it works, worker seven sends heartbeats that extend its lease.
The lease is what makes this safe from crashes. If worker seven dies, its heartbeats stop, the lease runs out, and the scheduler puts the run back to pending, where another worker claims it. Nothing is lost, and nobody has to notice. It's the visibility timeout from lesson fifteen, and the heartbeats from lesson twenty-nine, stored in the database.
At least once
Leases bring back an old friend: duplicates.
Worker seven is halfway through the payouts when it's paused for ten minutes: a long garbage collection, or a frozen virtual machine. Its five-minute lease runs out, and worker nine claims the run and starts the payouts again. Then worker seven wakes up, and carries on. Two workers are now paying the same restaurants. A scheduler can promise each run at least once, but never exactly once.
So the jobs themselves must make repeats harmless. They're idempotent, from lesson seventeen: running one twice has the same effect as once. A payout uses an idempotency key made of the restaurant and the date, from mock interview five, so the second payment is refused. And each claim carries a fencing token, from lesson twenty-eight, a number that grows with every claim, so writes from worker seven's older claim are rejected.
Priorities, retries and midnight
With ten million jobs a day from dozens of teams, the scheduler also has to be fair, and calm.
Priorities: when workers are busy, payouts go before the hourly report. Retries: a failed run is retried with backoff, up to its limit, then moved to dead letters, and its team is alerted, so a broken job is noticed before the morning. And a quota per team, so one team's runaway job, scheduling a million runs by mistake, can't starve everyone else.
And one very human problem: everyone schedules their daily jobs for exactly midnight. Ten minutes either side, the system is idle; at midnight, a million runs are due at once. So the scheduler spreads out jobs that don't need an exact minute, adding a random delay, like the jitter from lesson twenty-one, and keeps exact times for jobs that truly need them.
In the interview
Design a job scheduler, or design cron, is a senior question about leases, duplicates and fairness.
Say it like this: job definitions and individual runs live in a database, with runs indexed by their start time. Workers claim due runs with a conditional write and a lease, extended by heartbeats, so a dead worker's runs return to pending. Runs happen at least once, so jobs are idempotent, with fencing tokens. And priorities, retries and per-team quotas keep it fair.
Then expect these. A worker dies mid-run? Its lease runs out, and the run is claimed again. Can a job run twice? Yes, in rare cases, so jobs must be idempotent. And a million jobs at midnight? Spread the flexible ones with jitter, and add workers for the busy hour.
The common mistake is promising that every job runs exactly once. With leases, crashes and paused machines, the honest promise is at least once, with idempotent jobs making the repeats harmless. Say that, and you sound like someone who has been paged at midnight.
Quick check
A paused worker's lease expired, and another worker reran the job. What stops double payouts?
Show the answer
Idempotent jobs: a payout key per restaurant and date, plus fencing tokens.
Ten million jobs are all scheduled for exactly midnight. What's the problem, and the fix?
Show the answer
A spike at 00:00: spread jobs that don't need an exact time.
Remember: jobs and runs, with runs indexed by their start time. Workers claim runs with a conditional write and a lease, extended by heartbeats. And runs happen at least once, so jobs are idempotent. Tomorrow: counting billions of ad clicks, exactly, as they happen.
Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.