The whole course ▾

System Design · Lesson 29 of 39 · Week 6

Big files and long jobs

Updated 22 Oct 2026 · By Akshay, software engineer, India · 5 min read

Restaurants upload 2 GB menu videos over shaky networks. One drop at 90 percent, and it starts again. Today: big files.

The video for this lesson is coming soon.

New System Design videos go up at 6 pm. Subscribe on YouTube so you don't miss it.

Got a doubt? Ask in the comments under the video.Subscribe

QuickBite in this lesson

Restaurants upload menu videos to QuickBite.

QuickBite is the food delivery app we grow through the course, from one server to millions of orders.

Two gigabytes, one try

Two gigabytes, one try
Restaurant
app
App server
Object
storage
may I upload?
1
presigned URL
2
PUT 2 GB
3
~30 min on mobile data
drop at 90%: start over

In lesson nine, restaurant photos went straight to object storage through presigned URLs. Photos are a few megabytes. Menu videos are a different beast.

A restaurant films a two-minute tour of its kitchen, about two gigabytes, and uploads it from a phone. Same pattern as lesson nine: the restaurant app asks the app server for a presigned URL, then sends the whole file straight to object storage, in one single request.

At a phone's upload speed, two gigabytes takes nearly half an hour. Somewhere in that half hour, the connection drops: a lift, a tunnel, a switch from wifi to mobile data. A single upload request can't continue from where it stopped, so the app starts again from zero. Some restaurants never manage to upload a video at all. And even once it's there, the video needs minutes of work: smaller versions for phones, a thumbnail, and a check for anything inappropriate.

Upload in parts

Upload in parts
Multipart upload
1
Start: get an upload ID
2
Send 10 MB parts, a URL each
3
A part fails? Retry just that part
4
Complete: storage joins the parts
RESUMABLE
Theapprememberswhich
partsaredone:aftera
drop,itcontinuesfrom
part181,notpart1.

The fix for big uploads is to stop treating them as one thing.

Object storage supports multipart uploads, built for exactly this. The app starts an upload and gets an upload ID. The server presigns a URL for each part, say ten megabytes each, so a two-gigabyte video is about two hundred parts. The app sends the parts, even several at once, which is faster. If one fails, it retries just that part: ten megabytes, not two gigabytes. And when every part is in, the app calls complete, and storage joins them into one file. Anything over a few hundred megabytes should be uploaded this way.

And because the app remembers which parts finished, a dropped connection costs one part. Close the app, reopen it tomorrow, and it carries on from part one hundred and eighty-one. Storage can also list the parts it already has, so the server can tell a confused app exactly where to resume. The half-hour gamble becomes steady progress.

The video isn't ready yet

The video isn't ready yet
Object
storage
Job
queue
Video
workers
Jobs table
state and
progress
Restaurant
app
upload
complete
1
process
video
2
3 sizes, thumbnail
3
progress
60%
4
status?
JOBS ARE RECORDS
Everylongjobisa
recordwithastate:
queued,running,done,or
failed.

The upload is done. Now the video needs minutes of processing, and no request can wait that long.

When the upload completes, object storage announces the new file, and a job lands on a job queue: process video two-four-seven. Video workers take jobs off the queue and do the slow work: three sizes for different phones, a thumbnail, and a content check. They save the results back to object storage, and record progress in a jobs table: queued, processing, sixty percent, done. The restaurant app shows that progress, checking every few seconds, or through a server-sent event stream from lesson eighteen.

The key idea: every long job is a record with a state, not just a message in flight. Queued, running, done, or failed, with its progress and its error. Support can look it up, the app can show it, and nothing is ever lost in between. It's the same idea as the payment states in mock interview five.

When a worker dies mid-job

When a worker dies mid-job
Reliable long jobs
1
Heartbeats: still working
2
Timeout longer than the job
3
Checkpoints: resume, don't restart
4
Outputs safe to write twice
LIMITS
Everyjobgetsatime
limitandaretrylimit;
repeatedfailuresgotoa
dead-letterqueue.

A worker can die halfway through a ten-minute job. Lesson fifteen's tools still apply, with a few extras for long work.

Workers send heartbeats while they work, so a silent one is noticed, and its job handed to another. The queue's visibility timeout must be longer than the job, or extended as it runs, otherwise the job reappears while it's still running, and two workers do it at once. Long jobs save checkpoints, like sizes one and two are done, so a retry resumes instead of restarting. And outputs must be safe to write twice: the same file, under the same name.

And every job gets limits: a maximum running time, after which it's stopped and retried, and a maximum number of retries, after which it moves to a dead-letter queue for a person to inspect. A corrupt video file shouldn't keep a worker busy all night. Limits turn a stuck job into a visible one.

Do it now, or later?

Do it now, or later?
Inside the request
The user waits for minutes
A timeout kills the work
Web servers tied up
Job queue and workers
The user gets an answer at once
Progress shown while it runs
Workers scale on queue length
SCALE ON THE QUEUE
Addworkerswhenthe
queuegrows;batchwork
canrunoncheaper,
interruptiblemachines.

So why not just process the video inside the upload request? Here's the comparison.

Processing inside the request means the restaurant waits for minutes, a timeout or a dropped connection throws the work away, and web servers that should answer quick requests are tied up with video. A job queue and workers means the restaurant gets an answer at once, sees progress as it runs, and workers scale up and down with the length of the queue, separately from everything else.

That last point saves real money. Add workers when the queue grows, and remove them when it's empty. And because jobs checkpoint and retry, they can run on cloud machines that may be interrupted at short notice, which cost far less. A design that tolerates failure gets cheaper machines as a reward.

In the interview

In the interview
SAY THIS IN THE INTERVIEW
Multipart,resumableuploads
straighttostorage;
processingasjobsonaqueue,
withprogressandcheckpoints.
Follow-ups to expect
->
The upload drops at 90%?
->
A worker dies mid-job?
->
How does the user see
progress?
THE COMMON MISTAKE
Sendingbigfiles
throughyourservers,
orprocessingthem
insidetherequest.

Big files and long jobs come up in YouTube, Dropbox, photo apps, reports, and exports of every kind.

Say it like this: uploads go straight to object storage as multipart, resumable uploads with presigned URLs, and processing runs as jobs on a queue, with a state per job, heartbeats, checkpoints, retries and a dead-letter queue. Then say how the user sees progress.

Then expect these. The upload drops at ninety percent? It resumes from the last finished part. A worker dies mid-job? Missed heartbeats hand the job to another worker, which resumes from the last checkpoint. And how does the user see progress? From the jobs table, polled, or pushed with server-sent events. And does the app server touch the file at all? Almost never: it only signs URLs and records jobs, which is why it stays small and fast.

The common mistake is one of two: piping big uploads through your own servers, or doing minutes of processing while an HTTP request waits. Both work in a demo, and both fall over with real users.

Quick check

Quick check
?
A 2 GB file is sent in 10 MB
parts. How many parts, and
what's resent if one fails?
About 200 parts; only the
failed 10 MB part is resent.
?
Jobs take 10 minutes, but the
queue's visibility timeout is 30
seconds. What happens?
Duplicates: the job reappears
while running, and another
worker starts it too.
Remember
Big uploads: multipart and
resumable
Long work: a queue of jobs
with state
Heartbeats, checkpoints, safe
retries
Quick check

A 2 GB file is sent in 10 MB parts. How many parts, and what's resent if one fails?

Show the answer

About 200 parts; only the failed 10 MB part is resent.

Quick check

Jobs take 10 minutes, but the queue's visibility timeout is 30 seconds. What happens?

Show the answer

Duplicates: the job reappears while running, and another worker starts it too.

Remember: big uploads are multipart and resumable. Long work runs as jobs on a queue, each with a state. And reliable jobs use heartbeats, checkpoints, and outputs that are safe to retry. That's week six: feeds, search, nearby, contention, and big jobs. Tomorrow, the revision.

Practise with flashcardsYour progress stays on this device.
New System Design videos at 6 pm

Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.