The whole course ▾

System Design · Lesson 36 of 39 · Week 8

Design a distributed job scheduler

Updated 22 Oct 2026 · By Akshay, software engineer, India · 5 min read

QuickBite runs ten million scheduled jobs a day: payouts at midnight, reminders at noon, reports every hour. Today: a distributed job scheduler.

The video for this lesson is coming soon.

New System Design videos go up at 6 pm. Subscribe on YouTube so you don't miss it.

Got a doubt? Ask in the comments under the video.Subscribe

One cron machine

One cron machine
Cron server
one machine, one crontab
Payouts
00:00 daily
Reminders
12:00 daily
Reports
every hour
run
run
run
no payouts tonight

Every company has work that must happen at a certain time: pay the restaurants at midnight, remind customers about their saved basket at noon, build the sales report every hour. At first, one machine does it all.

That machine is the cron server: one machine, with one list of scheduled commands, called a crontab. At midnight it starts payouts. At noon, reminders. Every hour, reports. Simple, and for a small company, perfectly good. But QuickBite now has ten million scheduled jobs a day, from dozens of teams, and some of them move money.

Here's what goes wrong. At eleven fifty-nine, the cron server's disk fails, and it dies. Midnight passes, nobody notices, and two hundred thousand restaurants aren't paid. And even on a good day, one machine can't start ten million jobs, can't share them across many workers, and can't tell anyone whether a job ran, failed, or ran twice. We need cron, for a whole company.

Jobs and runs

Jobs and runs
Jobs
what, schedule,
retries, owner
Runs
job, run at, status,
lease
1 : many
Finding what's due
1
Index runs by their start time
2
Every second: runs due by now?
3
Due ones are handed to workers

First, the data model, because it splits the problem neatly in two.

A job is the definition: what to run, its schedule, like every day at midnight, how many retries it gets, and which team owns it. A run is one execution of that job at one moment: the midnight run of the twenty-ninth of November, its status, and who's working on it. Every time a run is created, the next one is worked out from the schedule and added too, so the runs table always holds what's due next.

Finding what's due is a simple query, made fast by lesson ten: index the runs by their start time. Every second, a scheduler asks: which pending runs are due by now? It gets back a small batch, and those are handed to workers. The scheduler itself does almost no work. And it can be several machines at once, because, as you'll see next, a run can only be claimed once.

Claiming a run

Claiming a run
Worker
7
Worker
9
Runs table
payout, 00:00:
pending
claim, if pending
1
claim, if pending
1
heartbeat
2
LEASES
Aworkerownsarun
onlyuntilitsleaseruns
out;adeadworker'srun
goesbacktopending.

Many workers, many due runs. Each run must go to exactly one worker, and must not be lost if that worker dies.

Two workers both see the midnight payout run as due, in the runs table. Each tries to claim it with a conditional write, from lesson twenty-eight: set status to running, worker to me, and a lease until five minutes from now, where the status is still pending. Exactly one write succeeds: worker seven gets it, and worker nine's write changes nothing. While it works, worker seven sends heartbeats that extend its lease.

The lease is what makes this safe from crashes. If worker seven dies, its heartbeats stop, the lease runs out, and the scheduler puts the run back to pending, where another worker claims it. Nothing is lost, and nobody has to notice. It's the visibility timeout from lesson fifteen, and the heartbeats from lesson twenty-nine, stored in the database.

At least once

At least once
Making repeats harmless
1
Jobs are idempotent: lesson 17
2
Payout key: restaurant + date
3
Fencing tokens on every write
Claiming a run
Worker
7
Worker
9
Runs table
payout, 00:00:
pending
claim, if pending
1
claim, if pending
1
heartbeat
2
paused 10 min, still
running
claimed after lease ran out
LEASES
Aworkerownsarun
onlyuntilitsleaseruns
out;adeadworker'srun
goesbacktopending.

Leases bring back an old friend: duplicates.

Worker seven is halfway through the payouts when it's paused for ten minutes: a long garbage collection, or a frozen virtual machine. Its five-minute lease runs out, and worker nine claims the run and starts the payouts again. Then worker seven wakes up, and carries on. Two workers are now paying the same restaurants. A scheduler can promise each run at least once, but never exactly once.

So the jobs themselves must make repeats harmless. They're idempotent, from lesson seventeen: running one twice has the same effect as once. A payout uses an idempotency key made of the restaurant and the date, from mock interview five, so the second payment is refused. And each claim carries a fencing token, from lesson twenty-eight, a number that grows with every claim, so writes from worker seven's older claim are rejected.

Priorities, retries and midnight

Priorities, retries and midnight
Running a fair scheduler
->
Priorities: payouts before reports
->
Retries with backoff, then dead letters
->
A quota per team
THE MIDNIGHT SPIKE
Everyoneschedulesjobs
for00:00:spreadthe
onesthatdon'tneedan
exactminute.

With ten million jobs a day from dozens of teams, the scheduler also has to be fair, and calm.

Priorities: when workers are busy, payouts go before the hourly report. Retries: a failed run is retried with backoff, up to its limit, then moved to dead letters, and its team is alerted, so a broken job is noticed before the morning. And a quota per team, so one team's runaway job, scheduling a million runs by mistake, can't starve everyone else.

And one very human problem: everyone schedules their daily jobs for exactly midnight. Ten minutes either side, the system is idle; at midnight, a million runs are due at once. So the scheduler spreads out jobs that don't need an exact minute, adding a random delay, like the jitter from lesson twenty-one, and keeps exact times for jobs that truly need them.

In the interview

In the interview
SAY THIS IN THE INTERVIEW
Jobsandrunsina
database,runsclaimed
withalease,idempotent
jobs,prioritiesandquotas.
Follow-ups to expect
->
A worker dies mid-run?
->
Can a job run twice?
->
A million jobs at midnight?
THE COMMON MISTAKE
Promisingexactly-once
runs:leasesand
pausesmake
duplicatespossible.

Design a job scheduler, or design cron, is a senior question about leases, duplicates and fairness.

Say it like this: job definitions and individual runs live in a database, with runs indexed by their start time. Workers claim due runs with a conditional write and a lease, extended by heartbeats, so a dead worker's runs return to pending. Runs happen at least once, so jobs are idempotent, with fencing tokens. And priorities, retries and per-team quotas keep it fair.

Then expect these. A worker dies mid-run? Its lease runs out, and the run is claimed again. Can a job run twice? Yes, in rare cases, so jobs must be idempotent. And a million jobs at midnight? Spread the flexible ones with jitter, and add workers for the busy hour.

The common mistake is promising that every job runs exactly once. With leases, crashes and paused machines, the honest promise is at least once, with idempotent jobs making the repeats harmless. Say that, and you sound like someone who has been paged at midnight.

Quick check

Quick check
?
A paused worker's lease expired,
and another worker reran the
job. What stops double payouts?
Idempotent jobs: a payout key
per restaurant and date, plus
fencing tokens.
?
Ten million jobs are all scheduled
for exactly midnight. What's the
problem, and the fix?
A spike at 00:00: spread jobs
that don't need an exact time.
Remember
Jobs and runs, indexed by
time
Claim with a lease; heartbeats
extend it
At least once, so jobs are
idempotent
Quick check

A paused worker's lease expired, and another worker reran the job. What stops double payouts?

Show the answer

Idempotent jobs: a payout key per restaurant and date, plus fencing tokens.

Quick check

Ten million jobs are all scheduled for exactly midnight. What's the problem, and the fix?

Show the answer

A spike at 00:00: spread jobs that don't need an exact time.

Remember: jobs and runs, with runs indexed by their start time. Workers claim runs with a conditional write and a lease, extended by heartbeats. And runs happen at least once, so jobs are idempotent. Tomorrow: counting billions of ad clicks, exactly, as they happen.

Practise with flashcardsYour progress stays on this device.
New System Design videos at 6 pm

Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.