The whole course ▾

System Design · Mock interview 20 of 20

Design ChatGPT: serving a large language model

Updated 22 Oct 2026 · By Akshay, software engineer, India · 9 min read

Design the serving system for ChatGPT.

Try it first

Before you read the interview, answer the question yourself. Give yourself 10 minutes: say your design out loud, or sketch it on paper. What would you ask? Which numbers matter? Which boxes, and what's the hard part? Then read on, and score yourself with the feedback at the end.

The video for this mock interview is coming soon.

New System Design videos go up at 6 pm. Subscribe on YouTube so you don't miss it.

Got a doubt? Ask in the comments under the video.Subscribe

Requirements

Requirements
Clarifying questions
->
Train the model too? -> No: serving only
->
Scale? -> 50M daily users
->
Answers? -> Streamed as they're written
->
Plans? -> Free and paid, with limits
Functional requirements
Send a message in a conversation
Stream the answer, token by token
Save and reopen conversations
Stop an answer midway
Non-functional requirements
->
First words within about a second
->
Steady streaming after that
->
GPUs kept busy: they're the cost
->
Graceful when demand beats supply
Out of scope
Training the model
Web browsing and tools
Images and voice
interviewer
Hi. Last one. Ready?
candidate
Ready. I'll draw as I go.
interviewer
Design the serving system for a chat assistant, like ChatGPT. Users type a message, and the answer appears word by word.
candidate
Okay. My guess is the scarce resource here isn't storage or bandwidth. It's GPU time. Let me ask about the model, the scale and the users.
interviewer
Go ahead.
candidate
Do we train the model too?
interviewer
No. It's trained already. We serve it.
candidate
What scale?
interviewer
About fifty million daily users, each sending around ten messages a day.
candidate
Are answers streamed as they're written?
interviewer
Yes, word by word.
candidate
And are there different plans?
interviewer
Free and paid, with different limits.
candidate
So, what it does. Send a message in a conversation, which carries the earlier messages as context. Stream the answer back, token by token; a token is a piece of a word, and the model produces one at a time. Save conversations, and reopen them later. And let users stop an answer midway.
interviewer
Why does stop matter?
candidate
I'll come back to that. It matters more than it sounds.
candidate
And how well. The first words appear within about a second, because waiting at a blank screen feels broken. Then steady streaming, faster than people read. GPUs kept busy, because they're the biggest cost, and an idle GPU is money burning. And graceful behaviour when demand is bigger than the GPUs we have.
candidate
I'll leave out training, web browsing and other tools, and images and voice. Does that match what you had in mind?
interviewer
Yes. Tell me how you'd use the GPUs well.

Estimation

Estimation
Back of the envelope
50M users × 10 = 500M messages a day
÷ 100,000 s = 5,000/s; peak ~ 15,000/s
× ~500 tokens out ~ 7.5M tokens/s
÷ ~5,000 tokens/s per server ~ 1,500 servers
WHAT THE NUMBERS DECIDE
GPUsarethescarce,expensive
resource:keepthembusywith
batching,andprotectthem
withqueuesandlimits.
candidate
Let me count messages, then tokens, then GPU servers.
interviewer
How much GPU capacity is this?
candidate
I'll work in tokens, because tokens are what GPUs produce.
candidate
Fifty million users, ten messages each, is five hundred million messages a day: five thousand a second, with a peak around fifteen thousand. Each answer is about five hundred tokens, so at the peak the system produces about seven and a half million tokens a second. If one GPU server, well batched, produces around five thousand tokens a second, that's about fifteen hundred GPU servers, each one very expensive.
interviewer
Are those numbers real?
candidate
They're illustrative. But the shape is real: GPU time is the bill.
candidate
So everything else, the web servers and the database, is cheap by comparison. GPUs are scarce, expensive, and slow to add. The design has two jobs: keep every GPU busy, which means batching, and protect them from more work than they can do, which means queues, priorities and limits.
interviewer
Good framing.

API and data

API and data
Conversations
user, title, model
Messages
role, text, token
count
Usage
user -> tokens
this minute
1 : many
limits
Endpoints
1
POST
/conversations/{id}/messages
2
<- a stream of tokens, as they're
made
3
POST …/stop: end it early
4
GET /conversations: history
candidate
Next, the API, with one unusual response, and a simple data model.
interviewer
What does the API look like, given that answers are streamed?
candidate
The response is a stream, not one object.
candidate
POST a message to a conversation. The response is a stream, server-sent events, sending each new token the moment it's generated, then a final event when the answer is done. POST to stop ends generation early. And GET the user's conversations, for the history sidebar, page by page.
candidate
Conversations: whose, a title, and which model. Messages: who said it, the user or the assistant, the text, and how many tokens it used, sharded by conversation, so a history loads from one shard. And usage: tokens per user, per minute and per day, in a fast store like Redis, for limits and billing.
interviewer
Why count tokens, not requests?
candidate
A long answer costs far more GPU time than a short one.

High-level design

High-level design
Chat
app
API
gateway
token limits
per plan
Chat
service
history,
prompt,
safety
Messages
DB
new
message
1
2
load
history
3
Scheduler
queue,
priority
by plan
GPU
servers
model in
GPU
memory
prompt
4
join a
batch
5
token, token…
6
streamed
7
Two phases per answer
1
Prefill: read the whole prompt at once
2
Decode: one token at a time
3
Prefill sets the wait for the first
word
4
Decode sets the streaming speed
TWO QUESTIONS
HowdoGPUsstaybusywhen
answerscomeonetokenata
time?Andwhathappenswhen
demandexceedstheGPUs?
interviewer
Okay. Show me the flow.
interviewer
Walk me through one message.
candidate
Everything around the GPUs first, then the GPUs themselves.
candidate
The chat app sends the message to the API gateway, which checks the user's plan and their token limits. The chat service loads the conversation's earlier messages from the messages DB, builds the full prompt, and runs a quick safety check on the input.
candidate
The prompt goes to a scheduler: a queue in front of the GPUs, which decides which request runs next, and on which server. GPU servers each hold the model in GPU memory, and generate tokens for many requests at once. Each token streams back to the chat service, and down the open stream to the app. When the answer finishes, the chat service saves it, and records the tokens used.
interviewer
Why a scheduler, and not just a load balancer?
candidate
A load balancer spreads requests evenly. The scheduler knows which GPU has free slots and memory, which matters far more here.
candidate
Inside a GPU server, each answer has two phases. Prefill: the model reads the whole prompt in one big, parallel step. Then decode: it produces the answer one token at a time, each step depending on the last. Prefill decides how long the user waits for the first word; decode decides the streaming speed.
interviewer
Good. Keep that in mind.
interviewer
So what's hard?
candidate
Two things. If decoding produces one token at a time per answer, how do I keep an expensive GPU busy, instead of mostly waiting? And what happens at the peak, when demand needs more GPUs than exist?
interviewer
Efficiency first.

Deep dive: keeping GPUs busy

Deep dive: keeping GPUs busy
One request at a time
Each step reads all the weights
For a single token of output
The GPU mostly waits on memory
Continuous batching
One step serves dozens of requests
Requests join and leave every step
Many times the tokens per GPU
THE CHOICE
Continuousbatching,with
theKVcachemanagedin
pages,andsharedprompt
prefixesreused.
High-level design
Chat
app
API
gateway
token limits
per plan
Chat
service
history,
prompt,
safety
Messages
DB
new
message
1
2
load
history
3
Scheduler
queue,
priority
by plan
GPU
servers
model in
GPU
memory
prompt
4
join a
batch
5
token, token…
6
streamed
7
dozens of answers per step
fill free slots every step
Two phases per answer
1
Prefill: read the whole prompt at once
2
Decode: one token at a time
3
Prefill sets the wait for the first
word
4
Decode sets the streaming speed
TWO QUESTIONS
HowdoGPUsstaybusywhen
answerscomeonetokenata
time?Andwhathappenswhen
demandexceedstheGPUs?
interviewer
Let's talk efficiency. It's cost.
interviewer
Decoding is one token at a time. How do you use the GPU well?
candidate
The key fact: one decode step for one request barely uses the GPU's processors. Most of the time goes on reading the model's weights from memory. So I'd share each step across many requests.
candidate
Serving one request at a time means each step reads all the weights to produce a single token, and the GPU mostly waits. Batching processes dozens of requests in each step, reading the weights once for all of them, so the output multiplies. And continuous batching goes further: requests join and leave the batch at every step, so a finished short answer frees its place immediately, instead of waiting for the longest one.
candidate
So, continuous batching. Its limit is memory. Each request keeps a KV cache, the model's working memory of everything it has read so far, which grows with every token. Managing that cache in small pages, as systems like vLLM do, wastes far less memory, so more requests fit in a batch.
interviewer
Anything else?
candidate
When many requests share the same start, like a system prompt, reuse its cached prefill instead of computing it again.
candidate
So each GPU server runs one batch, continuously, with dozens of answers in progress. The scheduler watches for free slots, as answers finish or users press stop, and fills them from the queue at the very next step.
interviewer
So that's why stop matters.
candidate
Yes. Stopping frees a slot, and its memory, for someone else, instantly.
interviewer
A user pastes a hundred-page document. What happens to everyone else on that GPU?
candidate
Its prefill is huge, and done in one step, every other answer in the batch stalls. So I'd split long prefills into chunks, interleaved with other requests' decode steps, and send very long prompts to a separate pool of servers set up for long contexts. One heavy request shouldn't freeze many light ones.

Deep dive: when demand beats supply

Deep dive: when demand beats supply
Graceful under overload
1
Admit only what the GPUs can take
2
Priority lanes: paid plans first
3
Limits in tokens per minute
4
Free plan: a smaller, faster model
LONG CONVERSATIONS
Historybeyondthemodel's
contextwindowistrimmed
orsummarised;shared
prefixesstaycached.
interviewer
Now, the peak. GPUs can't be added in minutes.
interviewer
At the busiest hour, demand is twice what the GPUs can serve. What do users experience?
candidate
Something has to give, so I'd decide in advance what gives, and for whom.
candidate
First, the scheduler admits only as much work as the GPUs can handle, and queues the rest with an honest message, instead of accepting everything and making every answer crawl. Second, priority lanes: paid plans ahead of free. Third, limits in tokens per minute, per user, so a few heavy users can't take everything. And fourth, when it's very busy, free users may be served by a smaller, faster model.
interviewer
What about very long conversations?
candidate
Every message re-sends the conversation so far, so a long chat makes every answer more expensive, and eventually exceeds the model's context window. So the chat service trims the oldest messages, or replaces them with a short summary, and keeps the system prompt and the recent turns. The unchanged start of the prompt can also be reused from the previous turn.
interviewer
A GPU server dies halfway through an answer. Then what?
candidate
The stream stops. The chat service notices, tells the app something went wrong, and offers to regenerate, which sends the prompt to another server. Saving the message is idempotent, keyed by the message ID, so a retried answer never appears twice in the history.
interviewer
How do you keep answers safe, when they're streamed before they're finished?
candidate
Check the input before generation starts. Then run a safety check over the output as it streams, in small chunks, and stop the stream if a chunk crosses a line. It adds a little delay per chunk, which is worth paying.
interviewer
Good.

Wrap-up

Wrap-up
Next steps
1
GPU capacity across regions
2
Faster decoding with a small draft model
3
Watch time to first token and cost per token
4
Compare models safely with experiments
INTERVIEWER'S FEEDBACK
Strong:GPUsasthescarceresource,
continuousbatching,KVcachecare,
admissionandpriorities.Nexttime:the
costperanswer.Verdict:stronghire.
interviewer
We're nearly at time.
interviewer
What would you improve with more time?
candidate
A few things.
candidate
First, GPU capacity across regions, routing requests wherever GPUs are free, so scarce capacity never sits idle in one place. Second, faster decoding with a small draft model that guesses several tokens ahead, which the big model checks in one step. Third, I'd watch the time to the first token, tokens per second, queue length, and cost per token. And fourth, compare new models safely, with experiments on a small share of traffic.
interviewer
Which would you do first?
candidate
The watching. Time to the first token is what every user feels.
interviewer
Okay, let's stop there. That was excellent. You saw straight away that GPUs are the scarce, expensive resource, and designed everything around them: continuous batching, careful use of the KV cache, a scheduler that admits only what it can serve, and priority lanes. One thing to add: put a number on what one answer costs to serve. I'd say a strong hire.
candidate
Thank you. That was a great one to finish on.
Interviewer's feedback

Strong: GPUs as the scarce resource, continuous batching, KV cache care, admission and priorities. Next time: the cost per answer. Verdict: strong hire.

Score yourself

Interviewers score these seven things. Give your own attempt a tick for each one you did.

  • Asks before drawing. Agrees what the system does, and how well: scale, speed, consistency.
  • Turns users into requests a second and bytes a day, and says what the numbers decide.
  • A small, clear contract, and a data model that fits how the data is read.
  • A working path end to end before any detail. Every box has a reason.
  • Picks the hard parts, compares two options, and chooses one with a trade-off said out loud.
  • What breaks, what happens then, and how users notice. Retries, duplicates, limits.
  • Thinks out loud, checks in with the interviewer, and takes a hint without losing the thread.

0 of 7

Your progress stays on this device.
New System Design videos at 6 pm

Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.