Designing ChatGPT

1.1What it has to do

Strip away images, voice, search and the rest, and the core of a chat product is a short list:

1. Accept a message in a conversation, from an app or a browser.
2. Generate a reply with a large language model, using everything said so far in that conversation.
3. Show the reply as it's written, word by word, and let the user stop it halfway.
4. Keep the conversation, so Arjun can come back to it tomorrow, edit a message, or ask a follow-up.
5. Share the hardware fairly, so one heavy user or one runaway script can't starve everyone else.

And the qualities it needs:

• Fast to start: the first words should appear within about a second. A model writes in small pieces of text called tokens (section 2 shows exactly what they are), so the wait from pressing Enter to the first piece is called time to first token (TTFT).
• Smooth while streaming: once words are flowing, each new piece should follow quickly enough that reading never stalls. That gap is called time per output token (TPOT).
• Affordable: in the Splitwise paper's 2023 cost table, one 8-GPU H100 machine rents for about $38 an hour, so the fleet has to squeeze as many answers out of each GPU as it can.
• Private and correct: Arjun's conversation must never show up in someone else's account, and must never be lost.

Uber's hard problem was finding nearby things in space. This one's is a resource: the GPU is very fast at arithmetic but short of memory, and most of this case study is about using it well.

1.2How big is it?

OpenAI's own research gives the scale. A study by OpenAI's economic research team, published through NBER in September 2025, reports that by July 2025 ChatGPT had about 700 million weekly users sending 18 billion messages a week, more than 2.5 billion a day, or roughly 29,000 messages a second on average. In February 2026 OpenAI announced 900 million weekly active users.

The unit that matters for the hardware, though, is the token. How many tokens a ChatGPT message involves isn't published. Probably the best public stand-in is a trace that Microsoft released from its Azure LLM inference services, captured in November 2023 and analysed in the Splitwise paper (ISCA 2024): for the conversation service, the median request had 1,020 prompt tokens going in and 129 tokens coming out.

2.1Text becomes tokens

A language model doesn't read letters or words. It reads numbers, each standing for a piece of text from a fixed vocabulary of about two hundred thousand pieces: common words, parts of words, punctuation, spaces glued to the word after them. Those pieces are the tokens, and the program that cuts text into them is a tokenizer. OpenAI publishes its tokenizers in an open-source library called tiktoken. This program cuts Arjun's question with o200k_base, the encoding tiktoken uses for GPT-4o and GPT-5 (install it with pip install tiktoken; the first run downloads the vocabulary):

import tiktoken
 
enc = tiktoken.get_encoding("o200k_base")   # what tiktoken uses for GPT-4o and GPT-5
msg = "Why is the sky blue? Explain it to a ten-year-old."
 
ids = enc.encode(msg)
print(f"{len(msg)} characters -> {len(ids)} tokens")
print(ids)
print([enc.decode([i]) for i in ids])
print(f"vocabulary size: {enc.n_vocab:,}")

50 characters -> 14 tokens
[13903, 382, 290, 17307, 9861, 30, 115474, 480, 316, 261, 4325, 8204, 12324, 13]
['Why', ' is', ' the', ' sky', ' blue', '?', ' Explain', ' it', ' to', ' a', ' ten', '-year', '-old', '.']
vocabulary size: 200,019

Fifty characters became 14 tokens, roughly 3.6 characters each. Notice that most tokens carry their leading space (' sky'), and that "ten-year-old" was cut into three pieces. Every cost in this chapter, from GPU memory to rate limits to the price of an API call, is counted in these units.

2.2One token at a time

Given a list of tokens, the model does one thing: it computes, for every token in its vocabulary, how likely that token is to come next. One run of the model over its input is called a forward pass. Inside, the tokens' numbers flow up through a stack of identical layers, dozens of them. Each layer has an attention step, where every token looks back at the tokens before it to pick up context, and a feed-forward step that transforms each token on its own. Both steps are mostly matrix multiplications against the model's learned numbers, its weights.

To write an answer, the system runs a forward pass on Arjun's 14 tokens (plus whatever instructions come before them), picks one likely next token, say Great, appends it, and runs the model again on the longer list to get the next one. This is called autoregressive generation: each output token becomes part of the input for the next. Arjun's answer of a few hundred tokens means a few hundred forward passes, one after another, and none of them can start until the one before has finished.

?Why can't the model write the whole answer at once?

Because token 40 of the answer depends on which token 39 turned out to be. The model only produces a probability for the next token, and the system has to choose one before it can ask about the token after. There's no way to compute the fifth word of a sentence before deciding the fourth, so generation is a loop, and its speed is the time of one trip around it.

2.3The KV cache

Look inside the attention step again. To process token 40, every layer compares it with a key vector for each of tokens 1 to 39, and blends their value vectors. A key and a value are lists of numbers that each layer computes from each token. They depend only on the token and the tokens before it, so once computed they never change.

A naive loop would recompute keys and values for all 39 earlier tokens at every step, which makes a 500-token answer cost roughly 500²/2 token-computations instead of 500. So every serving system keeps them: after computing a token's keys and values in every layer, it stores them in GPU memory and reuses them on every later step. That store is the KV cache. It makes each later step cheap in arithmetic, at the price of memory that grows with every token of the conversation. Section 6 will show that this memory, more than anything else, decides how many people one GPU can serve.

3.1The obvious design

Here's the simplest design that works: an API server receives Arjun's message, looks up his earlier messages in a database, builds the prompt, and hands it to a GPU server holding the model. It runs the loop until the model emits a special end-of-answer token, then returns the whole answer, which the API server saves and sends back to Arjun. One request at a time per GPU server, and a queue in front for everyone else.

To see how badly this does, we need numbers. OpenAI doesn't publish the sizes of its ChatGPT models, so we'll do the arithmetic on a published model of the same class, Meta's Llama 3 70B: 80 layers and about 70 billion weights. At 2 bytes per weight that's 140 GB, which doesn't fit in one 80 GB NVIDIA H100, so it runs on two. Chapter 45 works through this model on this hardware in detail; here we'll take its numbers and see what they mean for a chat service.

3.2Prefill and decode are different kinds of work

Arjun's request runs in two phases, and they behave very differently on a GPU.

Arjun's first forward pass processes the whole prompt. All of its tokens are known already, so the model can push them through every layer together: one weight matrix, read once from memory, multiplied against a thousand tokens' worth of numbers at once. This phase is called prefill, and it produces the first output token plus the KV cache for every prompt token.

Every pass after that processes exactly one new token, and this phase is called decode. Each weight is still read from memory, but now it's used for only one multiply and one add. Here is the arithmetic for both, taking Arjun's prompt as 1,000 tokens once the hidden instructions and history are counted, on two H100s (989 trillion floating-point operations, or FLOPs, a second of dense 16-bit math each, and 3.35 TB/s of memory bandwidth each, per NVIDIA's datasheet):

Prefill does about 1,000 operations for every byte it reads, so its time is set by arithmetic: it's compute-bound. Decode does about one operation per byte, so it spends 21 ms waiting for memory to deliver weights and 0.07 ms using them: it's memory-bandwidth-bound, and the GPU's arithmetic units sit idle roughly 99.7% of the time. Chapter 45's roofline section explains why the crossover between the two is near 300 operations per byte on an H100.

Arjun's answer of about 300 tokens therefore takes 71 ms of prefill and then about 300 × 21 ms ≈ 6.3 seconds of decode, and version 1 shows him nothing for all of it. Meanwhile the hardware he's using is doing useful arithmetic well under 1% of the time. Those are the two problems to fix, and the first is the easier one.

4.1Send each token as soon as it exists

Streaming doesn't make the answer any faster, but it feels completely different. Arjun's first token exists after 71 ms of prefill. If we send it to Arjun right away, and every later token the moment it's decoded, he starts reading within a fraction of a second, and at about 48 tokens a second the text arrives faster than he reads. So the user-facing latency becomes TTFT, and TPOT only has to beat reading speed.

Over HTTP, the usual way to do this is server-sent events (SSE): the client makes one ordinary HTTP request, and the server keeps the response open, writing a small text event each time it has something new. OpenAI's API documents exactly this for its streaming mode: with stream=true, the response arrives as server-sent events, starting with response.created, then a stream of response.output_text.delta events, each carrying a few characters, then response.completed. On the wire, each event is a couple of plain text lines followed by a blank line:

event: response.output_text.delta
data: {"type":"response.output_text.delta","delta":"Great question!"}
 
event: response.output_text.delta
data: {"type":"response.output_text.delta","delta":" Sunlight looks white"}

SSE suits this job because it's plain HTTP, so proxies and load balancers already handle it, and one-way, enough for an answer; the next message is a new request. The gateway also turns token ids back into text before sending, holding back any token that ends partway through a multi-byte character.

4.2Stop generating, and cancellations

Streaming adds a button: Stop generating. If Arjun sees that the answer is going the wrong way and presses it, the browser closes the stream. That close has to travel all the way to the model server and free Arjun's place there, or the GPU will keep decoding a few hundred tokens nobody will read. So every layer in the path has to handle cancellation as an everyday event.

Cancellation is also where a subtle class of bugs lives. On 20 March 2023, ChatGPT went offline after some users saw the titles of other users' conversations. OpenAI's write-up traced it to its Redis cache client, redis-py, used with Python's asyncio. If a request was cancelled after it had been sent to Redis but before its reply was read, the connection went back into the shared pool with that reply still in it, and the next request on that connection, belonging to someone else, received it. A server change that morning had sharply increased cancellations. That bug also exposed payment details of 1.2% of ChatGPT Plus subscribers active during a nine-hour window.

Streaming fixed what Arjun sees. It did nothing for the other problem: the GPU is serving one conversation and wasting more than 99% of its arithmetic doing it, while thousands of other people wait in the queue.

5.1Read the weights once, use them many times

Decode is slow because every step reads 140 GB of weights to produce one token. But nothing says those weights can only be used for one conversation. If 32 conversations are each waiting for their next token, the GPU can read each weight once and multiply it against 32 tokens' worth of numbers, one from each conversation. That's called batching, and for decode it's nearly free: the step takes a little longer because each conversation's KV cache must be read as well, but it produces 32 tokens instead of one. Chapter 45's arithmetic for this model puts it at about 24 ms per step for batch 32 against 21 ms for batch 1: about 1,330 tokens a second instead of 48, 28 times the throughput.

Look back at the estimate in section 1.2. At 3.7 million output tokens a second, serving each conversation alone at 48 tokens a second would take roughly 78,000 pairs of H100s, 155,000 GPUs, just for decode. At batch 32 it's roughly 2,800 pairs. (Real models, hardware and traffic differ; the ratio is the point.) Batching is where the economics of the whole service come from.

One way to batch is the one used for most machine-learning serving: collect requests into a group, run the group from start to finish together, then take the next group. This is request-level (or static) batching, and it has a flaw that shows up immediately with chat.

5.2Continuous batching

A fix came from a 2022 paper, Orca (OSDI 2022, from Seoul National University and FriendliAI). Its observation was that generation is a loop of forward passes, so the scheduler can make its decision at every pass instead of once per request. After each step, any conversation that has finished leaves the batch, and a waiting request takes its slot for the very next step. Orca called this iteration-level scheduling; today it's usually called continuous batching.

Mixing a prefill of 1,000 tokens with single decode tokens in one pass raises a technical problem: the inputs have different shapes, and GPU kernels like rectangular batches. Orca's second idea, selective batching, handles it. Most of a layer (the weight multiplies, the normalisations) treats each token independently, so all the tokens from all the requests can be flattened into one long list and multiplied together. Only attention needs to know which tokens belong to which conversation, so it's run per request. Orca's evaluation on a GPT-3-sized 175-billion-weight model reported 36.9 times the throughput of NVIDIA's FasterTransformer at the same latency.

5.3When a long prompt joins

Continuous batching created a new problem, and it's visible in the Scene above. When Arjun joined, that step had to push 1,000 prompt tokens through the model, about 71 ms of arithmetic, where an ordinary decode step takes about 21 ms. Sam and Li each waited three or four times longer than usual for their next token. Now imagine someone pastes a 30,000-token document: a prefill of a couple of seconds, during which every conversation in the batch freezes mid-sentence.

So prefill and decode interfere with each other. Prefill is compute-bound and wants big chunks of tokens; decode is memory-bound and wants short, regular steps. The Splitwise authors saw this in Azure's production traces: running both on the same machine "often leads to inconsistent end-to-end latencies".

One fix is to cut the prompt into pieces. Chunked prefill, introduced by Sarathi-Serve (OSDI 2024, from Microsoft Research India and Georgia Tech), splits a long prefill into chunks of a few hundred tokens and adds one chunk to each step alongside the running decodes, so no step is much longer than the others. Arjun's first token arrives a little later, because his prefill is spread over several steps, but nobody else's stream stalls. The paper reports 2.6 times the serving capacity of vLLM for Mistral-7B on one A100 under tail-latency limits.

Batching more conversations would give more throughput still, so why stop at 32? Because each conversation in the batch keeps its KV cache in GPU memory the whole time it's running, and that memory runs out first.

6.1How big is one conversation's KV cache?

Attention in each layer is split into parallel copies called heads, each with its own keys and values. Modern models let several heads share one set of keys and values, a trick called grouped-query attention that shrinks the cache: Llama 3 70B has 64 heads asking questions (query heads) but stores only 8 key/value heads per layer. Every layer stores a key and a value per key/value head for every token, so the cache per token is easy to work out from a model's published shape: 2 (key and value) × layers × key/value heads × head size × 2 bytes per number.

This program works out the per-token size for three models, then asks how many of Arjun's conversations fit on our two GPUs:

# KV cache bytes per token = 2 (a key and a value) x layers x KV heads x head size x bytes per number
def kv_bytes_per_token(layers, kv_heads, head_dim, bytes_per_num=2):
    return 2 * layers * kv_heads * head_dim * bytes_per_num
 
models = {
    # name: (layers, KV heads, head size), from each model's published config
    "OPT-13B (vLLM paper)":  (40, 40, 128),
    "Llama 3 70B":           (80, 8, 128),
    "gpt-oss-120b, full-attention layers": (18, 8, 64),
}
for name, cfg in models.items():
    print(f"{name:37} {kv_bytes_per_token(*cfg) / 1024:6.0f} KiB per token")
 
# Llama 3 70B in BF16 on two 80 GB H100s: 140 GB of weights, ~20 GB left over
per_token = kv_bytes_per_token(80, 8, 128)
free = 20e9
print(f"\nKV tokens that fit in 20 GB: {free / per_token:,.0f}")
 
conversation = 1_020 + 129          # median prompt + median reply, Azure trace 2023
print(f"one median conversation: {conversation} tokens, {conversation * per_token / 1e6:.0f} MB")
print(f"conversations at once, packed tightly:       {free / (conversation * per_token):.0f}")
print(f"conversations at once, reserving 8,192 each: {free / (8_192 * per_token):.0f}")

OPT-13B (vLLM paper)                     800 KiB per token
Llama 3 70B                              320 KiB per token
gpt-oss-120b, full-attention layers       36 KiB per token
 
KV tokens that fit in 20 GB: 61,035
one median conversation: 1149 tokens, 377 MB
conversations at once, packed tightly:       53
conversations at once, reserving 8,192 each: 7

Read the three lines at the top first. OPT-13B, the model the vLLM paper used for its example, has no head sharing, so each token costs 800 KiB; the paper notes that a single 2,048-token request could need 1.6 GB. Llama 3 70B is five times bigger but needs less per token, thanks to grouped-query attention. OpenAI's own open-weight model, gpt-oss-120b (released in August 2025), goes further: half of its 36 layers use full attention with small 64-number heads, and the other half only look back at the last 128 tokens, so those layers never hold more than 128 tokens' worth of cache. Model designers now shape attention around the serving cost of the KV cache.

Then the bottom half. On two H100s holding Llama 3 70B, about 20 GB is left after the weights (less in practice, since the forward pass needs working memory too). That's room for roughly 61,000 tokens of KV cache across all the conversations in the batch, and a median conversation from the Azure trace takes 377 MB of it. Packed tightly, 53 conversations fit. If the server instead sets aside room for each conversation's maximum length up front, say 8,192 tokens, only 7 fit, and batch 7 gives a fraction of batch 53's throughput.

6.2Why reserving the maximum wastes most of the memory

Reserving up front is what serving systems did before 2023, for a simple reason: GPU frameworks expect each tensor in one contiguous block of memory, and a conversation's KV cache grows by one token per step to a length nobody knows in advance. An easy way to give it a contiguous region is to allocate the maximum it could ever need.

The researchers behind vLLM (SOSP 2023, from UC Berkeley) measured what that costs. In existing systems, only 20.4% to 38.2% of the memory set aside for KV caches held actual tokens' keys and values. The rest was lost three ways:

• Reservation: space held for tokens that haven't been generated yet. Arjun's answer will be 300 tokens long, but the space for token 300 is held from the first step.
• Internal fragmentation: space for tokens that never come. The region was sized for 8,192 tokens; the conversation ends at 1,149.
• External fragmentation: gaps between regions of different sizes, too small for the next conversation's region, the same problem a malloc heap has (chapter 05).

It also measured how much this matters. On a 13-billion-weight model on a 40 GB A100, about 65% of GPU memory holds the weights and close to 30% holds KV caches, so the KV cache is the only part of memory that changes with load, and wasting two thirds of it caps the batch size and, with it, throughput.

So we need a way to give each conversation exactly as much KV memory as it has tokens, growing a little at a time, without needing it to be contiguous. Operating systems solved that problem for process memory decades ago.

7.1Blocks and block tables

Chapter 04 described how an operating system gives each process the illusion of one long contiguous address space: memory is cut into fixed-size pages, a process's pages can sit anywhere in physical RAM, and a page table maps each of the process's page numbers to the physical page that holds it. Pages are handed out only when first touched, and every page is the same size, so there are no awkward gaps.

vLLM applies the same idea to the KV cache, and its paper states the analogy outright: think of blocks as pages, tokens as bytes, and requests as processes. GPU memory for the KV cache is cut into fixed-size KV blocks, each holding the keys and values for 16 tokens (vLLM's default block size). A conversation's cache is a list of logical blocks, numbered 0, 1, 2…, and a per-request block table maps each logical block to a physical block somewhere in GPU memory, along with how many of its 16 slots are filled. A free list holds the unused physical blocks.

Walk through what disappeared. Reservation is gone, because blocks are allocated only when a token needs one. Internal fragmentation shrinks to at most one partly filled block per conversation, at most 15 tokens' worth. External fragmentation is gone, because every block is the same size, so any free block fits any request. In the paper's measurements, vLLM used 96.3% of KV memory for real token states, against 20.4% to 38.2% before, and that bigger batch gave 2 to 4 times the throughput of FasterTransformer and Orca at the same latency.

A whole batch is a set of these tables, one row per running request. When a new conversation joins, it gets a row; when one finishes, its row is reused:

7.2Attention over scattered blocks

There's a cost to all this, and it's in the attention computation. An ordinary attention kernel (a kernel is a function that runs on the GPU) expects a conversation's keys and values side by side in memory. Arjun's are now in blocks 7, 1, 12 and 3. So vLLM's attention kernel, PagedAttention, takes the block table as an extra input: for each logical block it looks up the physical block, fetches that block's 16 keys, computes attention scores against them, and moves on. It's a gather through an indirection table, much like a CPU walking a page table, except it's done in software inside the kernel.

vLLM's authors measured that indirection at roughly 20 to 26% more attention-kernel time than FasterTransformer's contiguous kernel. Attention is only part of each step, and the bigger batch the paging allows wins back far more than that. That's why every major open-source engine (vLLM, SGLang, TensorRT-LLM) now manages KV memory in blocks.

7.3Sharing blocks, and copy-on-write

Paging gives one more thing for free: two block tables can point at the same physical block. That's useful whenever two sequences start the same way. If Arjun's app asked for two alternative answers to the same prompt, both answers share the prompt's blocks and only diverge in the blocks after it. vLLM counts how many tables point at each physical block (a reference count), and if a sequence wants to write into a shared block, the manager first copies it and points that sequence at the copy. That's copy-on-write, exactly what an OS does for a process's pages after fork() (chapter 04).

In the paper's experiments, sharing saved 6.1% to 9.8% of KV memory when generating several samples per prompt, and 37.6% to 55.2% for beam search (which keeps many candidate continuations that share most of their tokens) on one dataset, and more on chat-style prompts. A much bigger win from sharing, though, comes from blocks that are identical across different requests, which is the subject of section 8.

7.4When the blocks run out

Blocks are allocated as conversations grow, so it's possible to admit a batch and then run out of free blocks halfway through everyone's answers. The engine must then preempt some conversation, take its blocks away, and give them back later. vLLM evicts all of a sequence's blocks or none of them, since attention needs every block of a sequence on every step, and it offers two ways to bring them back:

• Swap: copy the evicted blocks to CPU memory over PCIe, and copy them back when there's room. It's chapter 04's swapping, with CPU RAM in the role of the disk.
• Recompute: throw the blocks away, and when the conversation resumes, run its prompt and the tokens generated so far through one prefill to rebuild them. Since prefill is fast and compute-bound, this can be cheaper than the copies.

Users never see either, except as a brief pause in the stream. vLLM prefers to preempt the most recently arrived requests, so the ones that have been running longest finish first.

Paging lets one conversation's blocks be shared. The obvious next question is whether different conversations have blocks in common, and in a chat product they have a lot.

8.1Every request starts the same way

Look at what the model is given when Arjun sends his follow-up, Now make it shorter. A model has no memory between requests, so the prompt contains everything: the hidden instructions that ChatGPT puts before every conversation (called the system prompt; its length and content aren't published), then Arjun's first question, then the whole first answer, and only then the six new tokens. The Splitwise authors noted the same thing about chat APIs in general: the client sends the complete conversation so far, every time.

Left alone, our server would prefill Arjun's entire conversation again from scratch, recomputing hundreds of tokens' keys and values that it computed a minute ago. And the system prompt at the start is identical for millions of conversations, computed again for each of them.

So we cache KV blocks by content. A block's keys and values depend on its tokens and on every token before it, so its cache key has to cover the whole prefix. vLLM's automatic prefix caching does this by hashing each full block's tokens together with the hash of the block before it, so each hash stands for the whole prefix up to the end of its block. Two requests that share their first 64 tokens share their first four block hashes, so a lookup in a hash table from block hash to physical block finds the existing blocks, and prefill starts after them. Only full blocks are cached; a partly filled last block is computed fresh.

This program builds block tables that way for three requests: Arjun's first turn (prompt plus answer, everything his KV cache holds when it ends), a different user, Sam, with the same system prompt, and Arjun's second turn.

BLOCK = 16                       # tokens per KV block, vLLM's default
free = list(range(1000))         # physical block numbers on the GPU
refcount = {}                    # physical block -> how many requests use it
cache = {}                       # hash of (parent hash, tokens) -> physical block
 
def allocate(tokens):
    """Build a block table for a prompt, reusing cached full blocks."""
    table, parent, hits = [], 0, 0
    for i in range(0, len(tokens), BLOCK):
        chunk = tuple(tokens[i:i + BLOCK])
        key = hash((parent, chunk))
        if len(chunk) == BLOCK and key in cache:      # same prefix seen before
            block = cache[key]
            hits += 1
        else:
            block = free.pop(0)
            if len(chunk) == BLOCK:                   # only full blocks are shared
                cache[key] = block
        refcount[block] = refcount.get(block, 0) + 1
        table.append(block)
        parent = key
    return table, hits
 
system = list(range(100, 164))          # a 64-token system prompt, shared by everyone
question = list(range(500, 514))        # "Why is the sky blue? ..." (14 tokens)
answer = list(range(900, 950))          # the 50-token answer the model wrote
arjun_1 = system + question + answer    # everything in Arjun's KV cache after turn 1
sam = system + list(range(700, 720))    # Sam: same system prompt, different question
arjun_2 = arjun_1 + list(range(800, 806))   # turn 2 adds "Now make it shorter."
 
for name, prompt in [("Arjun, turn 1", arjun_1), ("Sam", sam), ("Arjun, turn 2", arjun_2)]:
    table, hits = allocate(prompt)
    print(f"{name:14} {len(prompt):3} tokens  blocks {table}  reused {hits}")
 
print("blocks shared by all three:", [b for b, n in refcount.items() if n == 3])

Arjun, turn 1  128 tokens  blocks [0, 1, 2, 3, 4, 5, 6, 7]  reused 0
Sam             84 tokens  blocks [0, 1, 2, 3, 8, 9]  reused 4
Arjun, turn 2  134 tokens  blocks [0, 1, 2, 3, 4, 5, 6, 7, 10]  reused 8
blocks shared by all three: [0, 1, 2, 3]

The tokens are stand-in numbers and the system prompt is made up, but the mechanism is the real one. Arjun's first turn fills eight blocks. Sam's request reuses the four blocks of the system prompt, and only needs new blocks for his own question. Arjun's second turn reuses all eight blocks of his first turn, so its prefill only has to process the last 6 tokens instead of 134. (The program never frees anything, so the reference counts just add up; a real engine would drop the count when a request finishes and keep the block cached until it's evicted.)

A finished request's blocks don't disappear when its reference count drops to zero. They go onto the free queue with their hashes intact, so if Arjun sends his follow-up a minute later and the blocks haven't been reused yet, they're found and revived. When the engine needs a block, it takes the one at the head of the queue, the least recently used, and deletes its hash from the table. vLLM even puts a finished request's blocks onto the queue in reverse order, so the last, most conversation-specific block is evicted before the shared beginning.

SGLang (2024, from Stanford and UC Berkeley) organises the same idea differently. Its RadixAttention keeps every cached prefix in a radix tree, a trie whose edges are labelled with runs of tokens, so finding the longest cached prefix of a new prompt is a walk down the tree, and eviction removes least-recently-used leaves. The paper reports up to 6.4 times higher throughput than earlier systems on workloads with heavy prefix reuse, such as multi-turn chat and few-shot prompts.

8.2Routing for cache hits

A prefix cache lives in the memory of one machine. Arjun's second turn only gets the benefit if it lands on the same machine as his first, and a load balancer that sends each request to the least busy server will scatter his turns across the fleet and miss almost every time.

OpenAI's API documentation describes how its own prompt caching handles this, and it's one of the few public descriptions of how OpenAI's serving fleet behaves inside. When the feature launched in October 2024, caching applied to prompts of at least 1,024 tokens, matched in 128-token steps, and cached input tokens cost half price. As of October 2026 the docs say:

• Requests are routed using a hash of the early tokens of the prompt, along with machine load and an optional prompt_cache_key the developer can set to group related requests.
• Cached state lives on individual machines, and traffic above about 15 requests per minute for one prefix can overflow to other machines, which then have to compute it again.
• For earlier models, cached prefixes typically stay active for 5 to 10 minutes of inactivity, up to an hour; an extended option keeps them up to 24 hours by storing encrypted key/value tensors in GPU-local storage.
• Caches are never shared between organisations, and cached tokens still count toward rate limits.

Every one of those rules follows from what we've built. Hash routing exists because the cache is per machine. Overflow exists because sending all traffic for a popular prefix to one machine would overload it. A minutes-long lifetime is what LRU eviction under memory pressure looks like. The 24-hour option moves blocks from GPU memory to slower local storage, a second tier like swapping. And keeping caches per organisation means one customer can't learn anything about another's prompts by timing cache hits.

Everything so far has assumed the model fits on one pair of GPUs. ChatGPT's models are larger than our 70-billion-weight example, and OpenAI doesn't publish by how much.

9.1Splitting the work across GPUs

We've already been cheating slightly: Llama 3 70B needs 140 GB for its weights, so it was split across two 80 GB GPUs from the start. GPT-3, at 175 billion weights, needs 350 GB. A frontier model can need far more. GPUs for this work are usually built and sold in groups of eight on one board, connected to each other by NVIDIA's NVLink, which on the H100 generation carries 900 GB/s per GPU, against 128 GB/s for the PCIe link to the host (both figures add the two directions together).

There are two basic ways to split a model across GPUs, and they cut it in different directions.

Tensor parallelism splits every layer. Each big weight matrix is cut into slices, one per GPU, and every GPU multiplies its slice for every token. The scheme most engines use comes from NVIDIA's Megatron-LM work (2019): the feed-forward block's first matrix is split by columns and its second by rows, so each GPU can do both multiplies on its own and only the final results need adding up across GPUs. That adding-up, called an all-reduce, happens about twice per layer per step, once after attention and once after the feed-forward block. For an 80-layer model that's about 160 all-reduces per decode step, each moving only about 16 KB per token (8,192 numbers of 2 bytes) but each one making every GPU wait for the slowest. That's tolerable at NVLink speeds and painful over a network, so tensor parallelism stays inside one board.

Pipeline parallelism splits the stack of layers instead. GPUs 0–7 hold layers 1–40, the next board holds layers 41–80, and each token's numbers are handed from one stage to the next once, a single small message per step.

?If pipeline parallelism communicates so little, why not use it everywhere?

Because it doesn't make one token any faster. Arjun's token has to pass through stage 1 and then stage 2, one after the other, so each stage is idle while the other works on his token. The pipeline only stays busy if several groups of requests are in flight at once, one in each stage, and the gaps when it can't fill them are called pipeline bubbles. Tensor parallelism does make one step faster, because every GPU works on every layer at once and each reads only its slice of the weights. So the usual layout, and the one Orca used for its 175-billion-weight model in 2022, is tensor parallelism inside an 8-GPU machine and pipeline parallelism across machines: 16 A100s as 2 pipeline stages of 8.

Newer hardware has moved the boundary. NVIDIA's GB200 NVL72 puts 72 GPUs in one liquid-cooled rack, all connected by NVLink, so the "inside one machine" domain for tensor parallelism is now a whole rack.

10.1Two workloads that want different machines

Chunked prefill (section 5.3) stopped long prompts from freezing everyone's streams, but it left prefill and decode sharing every GPU, and they want different things. Prefill is compute-bound and would like the most arithmetic it can get. Decode is bound by memory bandwidth and capacity, and would like lots of HBM. On a shared machine, one parallelism layout and one batch size have to serve both, and every prefill chunk still slows every decode step a little.

Two 2024 papers took the next step and put the phases on different machines. DistServe (OSDI 2024, from Peking University and UC San Diego) measured serving capacity as the request rate a GPU can handle while meeting targets for both time to first token and time per output token, and reported that splitting the phases served 7.4 times more requests, or met 12.6 times tighter latency targets, than colocated systems. Splitwise (ISCA 2024, from the University of Washington and Microsoft) used Azure's production traces and made a hardware argument: from the A100 to the H100, compute grew 3.43 times but memory bandwidth only 1.64 times and capacity not at all. Since decode can't use the extra compute, it can run on older or power-capped GPUs. Splitwise clusters reached 1.4 times the throughput at 20% lower cost, or 2.35 times the throughput for the same cost and power.

What it costs is moving the KV cache. Prefill builds Arjun's keys and values on one machine, and decode needs them on another.

A few milliseconds once per request is small next to the seconds of decode that follow, and Splitwise hides most of it by sending each layer's keys and values as soon as that layer's prefill finishes, while the next layer is still computing.

With the phases separated, decode is the part users wait on longest, a few hundred steps per answer, each one limited by memory bandwidth. One last trick in the serving engine attacks the number of steps itself.

11.1Checking is cheaper than writing

Recall where decode's time goes: at small batch sizes the GPU spends about 21 ms reading weights and a fraction of a millisecond computing. So a decode step that processes 5 tokens costs almost the same as one that processes 1, because the weights are read once either way. Decode can't use this, because it doesn't know token 2 until it has chosen token 1. But it could if someone guessed the next few tokens.

That's speculative decoding, published independently in 2022–2023 by Google (Leviathan, Kalman and Matias, ICML 2023) and DeepMind (Chen and colleagues, 2023). A small, fast draft model guesses the next few tokens, say 4. The big model then runs one forward pass over all 4 guesses together, shaped like a short prefill. That one pass tells it, for each position, what it would have produced itself. The system keeps the guesses up to the first one the big model disagrees with, takes the big model's own token at that position, and throws the rest away. If all 4 are accepted, the pass also yields a fifth token for free.

A modified rejection-sampling rule decides acceptance, and it makes the output follow exactly the same probability distribution as the big model alone would. Arjun gets exactly the answer he would have got; it only arrives sooner. Leviathan's paper reported 2 to 3 times faster generation on T5-XXL with identical outputs, and Chen's reported 2 to 2.5 times on DeepMind's 70-billion-weight Chinchilla.

?Why doesn't every request use it, then?

Because the trick spends spare arithmetic, and a busy server doesn't have much. At batch 1, decode leaves more than 99% of the GPU's arithmetic idle, so checking 5 tokens instead of 1 is almost free. At batch 64, the weights are already being reused 64 times per read, and attention over 64 KV caches adds real work, so verifying 4 extra tokens per conversation (most of which may be thrown away) starts to cost real time. Speculation helps most when latency matters more than throughput and the batch is small. Later designs drop the separate draft model and add extra prediction heads to the big model itself, as in Medusa, EAGLE and DeepSeek-V3's multi-token prediction, so the guesses come almost free with each step.

OpenAI's API exposes something of the same shape: with Predicted Outputs, a developer editing a file supplies the expected answer (mostly the old file) to speed up the response. The docs don't describe the mechanism, but they note that rejected prediction tokens are billed like generated ones.

That completes the serving engine: batched, paged, prefix-cached, split across GPUs and phases, speculating where it pays. Everything around it is about deciding which requests reach it, and remembering what was said.

12.1Count tokens, not requests

A web service usually limits each user to some number of requests per minute. Here that unit is nearly meaningless. Arjun's request had a 1,000-token prompt and a 300-token answer. Someone pasting a book chapter and asking for a long summary sends one request that costs a hundred times more prefill and holds KV memory for minutes. Two requests per minute can mean anything.

So the limits are counted in tokens. OpenAI's API documents limits on requests per minute and tokens per minute (and per day), any of which can be hit first. Before a request runs, the output length is unknown, so the limiter charges the larger of the request's max_tokens and an estimate from its character count. That's why the docs advise setting max_tokens close to what you expect. Responses carry headers such as x-ratelimit-remaining-tokens and x-ratelimit-reset-tokens. A request over the limit gets HTTP 429; and separately, when a model is overloaded as a whole, the API returns 503 with server_is_overloaded, and clients are told to retry with exponential backoff and jitter.

A natural way to build this is a token bucket per user or API key (chapter 40): the bucket refills at the allowed rate, each request takes out its estimated tokens up front, and the difference is refunded when the real count is known at the end. OpenAI's docs describe the limits and headers but not the algorithm behind them. ChatGPT's consumer limits (how many messages per few hours on each plan) work in the same spirit and change often, so we won't quote them.

12.2Picking a replica, and saying no

A router in front of the model servers now has several things to weigh for each request: which model it's for, which replica holds its prefix in cache (section 8.2), and which replicas have free KV blocks and short queues. Admitting a request to a replica that's out of KV memory doesn't fail; it causes preemptions (section 7.4) that slow everyone on that replica. So the router's load signal should be KV memory and queue depth, not CPU or request count.

When the whole fleet is full, something has to give, and it's better to refuse some requests quickly than to queue everyone until they time out. Traffic can jump far beyond any forecast. OpenAI's January 2026 post on its database describes the March 2025 launch of image generation in ChatGPT, when more than 100 million new users signed up within a week. A design for that day needs queues with bounded length, clear errors that tell clients to back off, and limits that can be tightened for free users first, so paying users and the API keep working.

13.1A conversation is a tree of messages

GPUs forget everything when a request ends (apart from whatever prefix blocks happen to survive in cache for a few minutes), so the conversation itself has to live in ordinary storage. When Arjun sends his follow-up, the conversation service loads his earlier messages to build the prompt, and when the answer finishes streaming, it saves the new pair.

A conversation isn't quite a list. ChatGPT lets you edit an earlier message or regenerate an answer and then flip between the versions, so one message can have several children, and the conversation the model sees is one path from the root to the latest message. A natural schema stores each message with its conversation, its parent, its role (user, assistant, tool) and its content, and keeps a pointer to the current leaf:

Rebuilding the prompt means walking from the current leaf up through the parents to the root, which takes one query on the partition when all of a conversation's messages are stored together. The partition key matters most: every read and write for Arjun's chat goes to one place, and different conversations spread evenly across the store (chapter 29).

13.2What OpenAI has published about its storage

Where OpenAI stores message contents is unpublished. What it has described, in a January 2026 post, is the PostgreSQL deployment behind ChatGPT: a single primary Azure PostgreSQL flexible server taking all writes, nearly 50 read replicas spread across regions, millions of queries a second, low double-digit-millisecond p99 latency and five-nines availability. To keep the single writer healthy, OpenAI moved "shardable, write-heavy workloads" to sharded systems such as Azure Cosmos DB, and requires new tables for new features to go there too. The post also describes the failure modes it designed against: a surge of cache misses sending a flood of reads to Postgres, handled with a lock so only one request per missing key refills the cache, and a write storm when the image-generation launch pushed write traffic up more than tenfold.

The 2023 incident in section 4.2 adds one more piece: ChatGPT used a Redis Cluster to cache user information in front of its database. Put together, the published picture is the classic one: a relational core for accounts and metadata, read replicas and a cache to absorb reads, and a horizontally sharded store for the data that grows with every message.

14.1Every box, and why it's there

14.2From top to bottom

15.1Failures this design has to survive

December 2024's row is worth a sentence more. On 11 December 2024 all OpenAI services were degraded or down for about four hours and twenty minutes. A new telemetry service made every node in each Kubernetes cluster run expensive calls against the cluster's API servers, the cost grew with cluster size, and the large clusters' control planes fell over. Model servers themselves were fine, but service discovery uses DNS, which depended on those API servers, so once cached DNS records expired, about twenty minutes later, services couldn't find each other. Engineers then couldn't reach the overloaded control plane to remove the change. Staging hadn't caught it because only the biggest clusters triggered it.

15.2The tradeoffs, in one table