Design ChatGPT: LLM Chat Application System Design

"Design ChatGPT" has become a standard system design question at product companies and AI startups, and it is a trap for candidates who treat it like "design WhatsApp with an API call". The chat part is the easy 20%. The interesting 80% is that the backend is compute-bound on scarce, expensive GPUs, responses take seconds to produce, a single request can be a hundred times heavier than another, and your cost per user is not a rounding error.

This walkthrough follows the order I recommend in an interview: requirements, numbers, API, data model, architecture, then the deep dives an interviewer will actually push on. If you are fuzzy on what happens inside the model itself, read how ChatGPT works for engineers first; here we treat the model as a black box that turns a prompt into a token stream.

Step 1: Clarify Requirements

Spend two or three minutes here. The scope of "ChatGPT" is enormous (images, voice, code execution, plugins), so pin down a core and say explicitly what you are leaving out.

Functional requirements

• A signed-in user starts a conversation, sends a message, and sees the assistant's reply stream in token by token.
• Conversations are persisted; the sidebar lists past conversations, and the user can reopen one and continue it.
• The user can stop generation mid-stream, regenerate a response, and edit an earlier message (which forks the conversation).
• Multiple models/tiers: a fast cheap model for free users, a larger model for paid users.
• Usage limits per plan (for example, N messages per few hours on the large model).

Non-functional requirements

• Time to first token (TTFT) under roughly 1 second at p50 and a few seconds at p99. This is the latency users feel most.
• Inter-token latency fast enough to read comfortably: 20 to 60 tokens/second per stream is fine for a human reader.
• High availability of the chat experience; graceful degradation (smaller model, queueing) rather than hard errors when GPUs are saturated.
• Durability of conversation history; it is user data.
• Cost efficiency: maximise tokens generated per GPU-second.
• Safety: input and output moderation, abuse prevention.

Out of scope (say it out loud): model training, image generation, voice, tool use and browsing. Mention that tools fit into the orchestrator later.

Step 2: Capacity Estimates (Show the Math)

These numbers are interview assumptions, not real OpenAI figures. The point is to show how token counts drive GPU count. If you want a refresher on the method, see back-of-the-envelope estimation.

• Daily active users: 50 million.
• Messages per user per day: 5 → 250M messages/day.
• Average: 250M / 86,400 ≈ 2,900 messages/s. Peak at 3x ≈ 9,000 messages/s.
• Average output length: 400 tokens. Average prompt (with history and system prompt): 2,000 tokens.

Token throughput

Peak output generation = 9,000 msg/s × 400 tokens = 3.6M output tokens/s. Peak prompt processing (prefill) = 9,000 × 2,000 = 18M input tokens/s. Prefill is processed in parallel across the prompt, so it is far cheaper per token in wall time than decode, but at these volumes it is not free.

GPU fleet

Assume one 8-GPU node serving a 70B-class model with continuous batching sustains about 3,000 output tokens/s aggregate (a rough, hardware- and engine-dependent assumption). Then 3.6M / 3,000 ≈ 1,200 nodes ≈ 9,600 GPUs at peak, before headroom and before prefill load. That number is why "just call the model" is not an answer.

Concurrency

At 40 tokens/s per stream, a 400-token answer takes about 10 s. By Little's law, concurrent streams ≈ 9,000/s × 10 s = 90,000 open streams at peak. Each needs an open HTTP connection on the edge and a KV cache slot on a GPU.

Storage

One message pair (user plus assistant) is roughly 600 tokens ≈ 2.4 KB of text plus ~0.5 KB metadata. 250M × ~3 KB ≈ 750 GB/day, about 270 TB/year before replication. Large but ordinary for a distributed database.

The takeaway to say aloud: storage and API servers are a solved problem; the GPU fleet is the system. Everything else exists to keep those GPUs busy and to protect them.

Step 3: API Design

Keep the public surface small. The interesting endpoint is the one that streams.

The stream itself is a sequence of SSE events:

event: meta
data: {"generation_id":"gen_8f2","message_id":"msg_77","model":"large-v3"}

id: 1
event: delta
data: {"text":"Consistent"}

id: 2
event: delta
data: {"text":" hashing maps"}

event: done
data: {"finish_reason":"stop","prompt_tokens":1874,"output_tokens":412}

Two details that interviewers like: the parent_message_id makes edit and regenerate the same operation (post a new child under a different parent), and the idempotency_key stops a client retry from starting a second, billed generation.

Step 4: Data Model

Conversations are append-heavy, read by key, and never joined across users. That points to a wide-column or key-value store (Cassandra, ScyllaDB, DynamoDB, Bigtable) partitioned by conversation. Postgres works at smaller scale; if you pick it, say how you would shard it (see sharding strategies).

-- Sidebar: one partition per user, newest first
CREATE TABLE conversations_by_user (
  user_id         uuid,
  updated_at      timestamp,
  conversation_id uuid,
  title           text,
  PRIMARY KEY ((user_id), updated_at, conversation_id)
) WITH CLUSTERING ORDER BY (updated_at DESC, conversation_id ASC);

-- Messages: one partition per conversation, ordered by time-sortable id
CREATE TABLE messages (
  conversation_id uuid,
  message_id      timeuuid,
  parent_id       timeuuid,      -- tree: edits and regenerations
  role            text,          -- system | user | assistant | tool
  content         text,
  model           text,
  prompt_tokens   int,
  output_tokens   int,
  status          text,          -- streaming | complete | cancelled | failed
  PRIMARY KEY ((conversation_id), message_id)
);

Why a tree, not a list

When a user edits message 3, messages 4 onward are not deleted; they become a sibling branch. Store parent_id on every message and a current_leaf_id on the conversation. To render, walk from the leaf up through parents. Conversations are short enough (hundreds of messages at most) that loading the whole partition and walking it in memory is fine.

Updating the sidebar

Because updated_at is part of the clustering key in conversations_by_user, "move to top" is a delete plus insert. That is acceptable at a few writes per conversation; alternatively keep the list in Redis sorted sets keyed by user and treat the table as the source of truth.

Persisting a streaming message

Do not write to the database on every token. Write the assistant message row with status = streaming when generation starts, buffer tokens in the stream service (and in Redis for resumption), and write the final content once at completion or cancellation. If the server dies mid-stream, a sweeper marks stale streaming rows as failed so the UI can offer "regenerate".

Step 5: High-Level Architecture

The control plane (Chat API, orchestrator, conversation store) is ordinary stateless services plus a database; it scales with CPUs and is cheap. The inference plane (gateway plus GPU pools) is where the money and the hard problems are. Keeping them separate lets you scale and deploy them independently: you will deploy the API 20 times a day, but rolling a model pool means draining tens of thousands of in-flight streams.

Request lifecycle

1. Accept. Chat API authenticates, checks the idempotency key, and reserves estimated tokens from the user's bucket.
2. Persist the user message. Write the user message and a placeholder assistant message (status = streaming).
3. Build the prompt. Orchestrator loads the active branch, applies the context budget (next sections), runs input moderation, and picks a model.
4. Schedule. Inference gateway admits the request into a model pool queue, prioritised by plan, and routes it to a server with free KV cache capacity, preferring one that already has this conversation's prefix cached.
5. Generate and stream. Tokens flow back from the model server through the gateway to the client over SSE; they are also appended to a Redis stream for resumption.
6. Finalise. On stop/finish/cancel, write the final message, reconcile token usage, emit a usage event to Kafka, and update the sidebar timestamp.

Deep Dive 1: GPU Inference Serving

This is the section that separates strong candidates. You do not need to know CUDA, but you must know the shape of LLM inference.

Prefill vs decode

Generation has two phases. Prefill processes the whole prompt in one parallel forward pass and builds the KV cache; it is compute-bound and determines TTFT. Decode generates output one token at a time, each step reading all the model weights and the KV cache from GPU memory; it is memory-bandwidth bound and determines tokens/second. A long prompt hurts TTFT; a long answer occupies the GPU for many sequential steps.

Continuous batching

Because decode is bandwidth-bound, running one request per GPU wastes most of the compute: reading 140 GB of weights to produce one token for one user is absurd when you could produce one token for each of 64 users with nearly the same memory traffic. Batching is mandatory. Naive static batching waits for every sequence in the batch to finish, so a 20-token answer is stuck behind a 2,000-token one. Continuous batching (also called in-flight batching) admits and evicts requests at every decode step. Serving engines like vLLM, Hugging Face TGI and NVIDIA TensorRT-LLM implement it, and it is the single biggest throughput lever.

KV cache is the real capacity limit

Each in-flight sequence keeps keys and values for every token in every layer. For a 70B-class model with grouped-query attention this is on the order of hundreds of KB per token, so a 4,000-token conversation can need over a gigabyte of KV cache. The number of concurrent streams a node can hold is (GPU memory − weights) / KV cache per sequence, not "number of requests the CPU can handle". PagedAttention (vLLM) allocates KV cache in fixed-size blocks to avoid fragmentation, which raises the achievable batch size. For a deeper treatment of these techniques, see LLM inference optimisation: KV cache, quantisation and batching.

Admission control and queueing

The inference gateway is not a dumb load balancer. It must:

• Track per-server free KV blocks and only route a request to a server that can fit its prompt plus expected output.
• Keep a priority queue per model pool: paid and API customers ahead of free users; interactive chat ahead of batch jobs.
• Shed or downgrade when queue wait exceeds a threshold: route free users to the small model, show "high demand" messaging, or reject with 429 and a Retry-After.
• Prefer prefix-cache affinity: send follow-up turns of the same conversation to the server that still holds its KV cache, so prefill only processes the new tokens. Use consistent hashing on conversation id with a load-aware fallback.

Prefill/decode disaggregation (good bonus point)

Large deployments sometimes run prefill and decode on separate GPU pools: prefill nodes crunch prompts and ship the KV cache to decode nodes. This stops a burst of huge prompts from stalling everybody's token stream. Mention it as an optimisation, not a baseline.

Model routing

Not every message needs the largest model. A lightweight classifier (or simple rules: plan, prompt length, presence of code) routes requests between pools. Routing even a third of traffic to a model that is several times cheaper changes the fleet size materially. Make routing decisions observable, because quality complaints will trace back to them.

Scaling the GPU fleet

GPUs cannot be autoscaled like stateless pods: nodes take minutes to boot and load tens of GBs of weights, and capacity may not be available on demand. The practical approach is a baseline reserved fleet sized for typical peak, predictive scaling ahead of known daily peaks, and load shedding for the rest. Batch workloads (evals, offline API jobs) fill troughs.

Deep Dive 2: Streaming Tokens with SSE

Why SSE rather than WebSockets? The interaction is request in, stream out. SSE is plain HTTP, passes through proxies and load balancers, multiplexes over HTTP/2, and has built-in reconnection semantics through Last-Event-ID. WebSockets earn their keep for voice or truly bidirectional sessions. The trade-offs are covered in WebSockets vs SSE vs long polling.

import json
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse

app = FastAPI()

async def sse(gen_id: str, token_iter, request: Request):
    yield f"event: meta\ndata: {json.dumps({'generation_id': gen_id})}\n\n"
    seq = 0
    async for text in token_iter:
        if await request.is_disconnected():
            await cancel_generation(gen_id)      # free the GPU slot
            return
        seq += 1
        await append_to_resume_log(gen_id, seq, text)   # Redis stream, TTL ~10 min
        yield f"id: {seq}\nevent: delta\ndata: {json.dumps({'text': text})}\n\n"
    yield "event: done\ndata: {}\n\n"

@app.post("/v1/conversations/{conv_id}/messages")
async def send(conv_id: str, request: Request):
    body = await request.json()
    gen_id, token_iter = await start_generation(conv_id, body)
    return StreamingResponse(
        sse(gen_id, token_iter, request),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
    )

Production details that matter

• Disable proxy buffering. Nginx and some load balancers buffer responses by default, which turns a stream into a single blob after 10 seconds. The X-Accel-Buffering: no header and proxy config fix this.
• Heartbeats. Send an SSE comment line (: ping) every ~15 s while queued so idle-timeout proxies do not cut the connection before the first token.
• Coalesce tokens. Flushing every single token creates many tiny frames. Flushing every 20 to 50 ms batches a few tokens with no visible difference.
• Cancellation must reach the GPU. If the user clicks Stop or closes the tab, propagate cancellation to the model server so it evicts the sequence. Otherwise you pay to generate tokens nobody reads.
• Resumption. Mobile networks drop. Writing each chunk to a short-lived Redis stream keyed by generation_id lets a reconnecting client call GET /generations/{id}/stream with Last-Event-ID: 57 and receive chunks 58 onward, while generation continues independently of the client connection.
• Decouple generation from the connection. The cleanest design has the model output go to the resume log, and every client connection (original or reconnected) tails that log. Then a dropped connection never wastes a generation, and multiple tabs can watch the same answer.

Deep Dive 3: Context Window Management

The model is stateless. "Memory" across turns is the orchestrator re-sending history every time, and every re-sent token costs prefill compute and money. A 50-turn conversation can easily exceed the model's context window, and well before that it becomes slow and expensive.

Prompt assembly within a budget

Give each request a token budget (say 8,000 tokens for the prompt, leaving room for the answer) and fill it in priority order:

1. System prompt (fixed, a few hundred to a few thousand tokens). Keep it identical across users so prefix caching can reuse it.
2. User-level memory or custom instructions, if the product has them.
3. The new user message, always included in full (or rejected if too long).
4. Recent turns verbatim, newest first, until the budget is nearly used.
5. A rolling summary of everything older, regenerated by a cheap model every N turns and stored on the conversation.
6. Retrieved snippets from older turns or uploaded files via embeddings, when the user refers back to something (the RAG pattern, covered in build a RAG system).

Counting tokens correctly

Use the model's own tokenizer, not character counts. Cache the token count of each stored message so budget calculation is a sum, not a re-tokenisation of the whole history on every turn.

Prefix caching

If turn 12 sends the same first 6,000 tokens as turn 11 plus a few hundred new ones, a server that kept turn 11's KV cache only needs to prefill the new tokens. This is why routing affinity matters and why you should append rather than rewrite history: if your summariser rewrites the start of the prompt every turn, you destroy the cacheable prefix. A good compromise is to re-summarise only occasionally, accepting a cache miss on those turns.

Deep Dive 4: Rate Limiting by Tokens

A classic per-request limiter (see design a rate limiter) is necessary but insufficient. One request with a 100k-token document costs as much GPU time as hundreds of "hi" messages. Limit on what actually consumes capacity: tokens per minute (TPM) alongside requests per minute (RPM), and for the consumer app, plan-level message caps on the expensive model.

Reserve, then reconcile

You do not know the output length up front. So:

1. Estimate cost = prompt tokens (known after assembly) + max_output_tokens (or a learned expected length).
2. Reserve that amount atomically from the user's token bucket. If it does not fit, return 429 with Retry-After.
3. Reconcile after completion: refund the unused part, or charge the overrun if you reserved an expected rather than maximum length.

-- KEYS[1] = bucket key
-- ARGV: capacity, refill_per_sec, now_ms, cost (negative cost = refund)
local cap  = tonumber(ARGV[1])
local rate = tonumber(ARGV[2])
local now  = tonumber(ARGV[3])
local cost = tonumber(ARGV[4])

local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(b[1]) or cap
local ts     = tonumber(b[2]) or now
tokens = math.min(cap, tokens + math.max(0, now - ts) / 1000 * rate)

local allowed = 0
if cost <= tokens then
  tokens = math.min(cap, tokens - cost)
  allowed = 1
end
redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil(cap / rate * 1000) + 1000)
return {allowed, tostring(tokens)}

import time
import redis

r = redis.Redis()
bucket = r.register_script(open("token_bucket.lua").read())

TPM = 200_000                      # plan limit: tokens per minute
RATE = TPM / 60.0                  # refill per second

def reserve(user_id: str, tokens: int) -> bool:
    allowed, _left = bucket(keys=[f"tpm:{user_id}"],
                            args=[TPM, RATE, int(time.time() * 1000), tokens])
    return allowed == 1

def reconcile(user_id: str, reserved: int, actual: int) -> None:
    # negative cost refunds unused tokens; positive charges an overrun
    bucket(keys=[f"tpm:{user_id}"],
           args=[TPM, RATE, int(time.time() * 1000), actual - reserved])

Note the tostring in the Lua return: Redis converts Lua numbers to integers, silently truncating fractional balances. Refunds always succeed because a negative cost is always less than or equal to the current balance, and the math.min(cap, …) keeps the bucket from overflowing.

Layers of limits

• Per user: TPM and RPM by plan; message caps on premium models over a rolling window.
• Per organisation / API key: TPM tiers that grow with spend.
• Global per model pool: the inference gateway's queue depth is itself a limiter. When it is full, shed load by priority.
• Abuse: anomaly detection on accounts generating continuously (scraping, resale of access).

Deep Dive 5: The Cost Model

Interviewers at AI companies increasingly ask "what does one message cost you?" Here is an illustrative calculation; GPU prices and throughput vary widely by provider, contract and engine, so treat every number as an assumption to state, not a fact.

At that scale, a 10% throughput improvement is worth more than most engineers' annual salaries, which is why the cost levers deserve their own list:

• Batching and KV cache efficiency: bigger effective batch, more tokens per GPU-second.
• Quantisation (FP8, INT8, INT4 weights): smaller weights mean less memory traffic per decode step and more room for KV cache, at some quality risk.
• Speculative decoding: a small draft model proposes several tokens, the large model verifies them in one pass.
• Prefix caching: stop re-paying prefill for system prompts and earlier turns.
• Routing to smaller models for simple queries.
• Output caps and conciseness in the system prompt: output tokens are the expensive ones.
• Exact-match response cache helps only for highly repeated prompts (FAQ-style starters); semantic caching is risky for a general assistant because similar questions often need different answers.

Bottlenecks, Failure Modes and Trade-offs

Consistency choices

Conversation history can be eventually consistent across regions: a user rarely writes to the same conversation from two continents at once. Billing and usage counters need exactly-once-ish accounting, so emit usage events with the generation id as a dedup key and aggregate idempotently downstream.

What Interviewers Look For

Typical follow-ups and how to answer briefly:

• "How would you add web browsing or tools?" The orchestrator runs an agent loop: model emits a tool call, orchestrator executes it in a sandbox, appends the result, and calls the model again. Stream intermediate status events. See AI agents.
• "How do you evaluate a new model before rollout?" Offline eval suites, then shadow traffic, then a canary with user feedback signals (thumbs, regenerate rate). See LLM evals.
• "How do you prevent prompt injection in uploaded files?" Treat retrieved content as untrusted data, isolate tool permissions, and require confirmation for side-effecting actions.
• "Multi-region?" Inference pools per region, global routing based on queue depth and latency, conversation data replicated asynchronously.

Common Mistakes in This Interview

• Rate limiting by requests only. Token-based limits are the expected answer.
• Writing every token to the database. Tens of millions of writes per second for no benefit. Buffer and write once.
• Using WebSockets without a reason. Not wrong, but you should justify it over SSE.
• Forgetting cancellation. Abandoned generations are pure waste at this scale.
• Ignoring context growth. Sending the full history forever makes cost grow quadratically over a long conversation (each turn re-sends all previous turns).
• Autoscaling GPUs like web pods. Capacity is slow and scarce; plan for reservation, prediction and shedding.
• No numbers. Without a token-throughput estimate you cannot justify a single design choice.

If you want the broader picture of how ML-heavy systems are evaluated in interviews, pair this with the AI system design interview guide and the AI app architecture post.

Frequently Asked Questions

Pranjal Jain

Ex-Microsoft SDE · IIT Kanpur · Founder of Prepflix. Helps engineers crack startup and product company interviews across India.