Topic 06: From nanoGPT to Production LLM Serving Engines
Overview: Connecting our prototype implementations in nanoGPT and CUDA FlashAttention to industrial-grade LLM inference engines (vLLM, SGLang, TensorRT-LLM): resolving memory fragmentation with PagedAttention, scaling dynamic prefix reuse with RadixAttention, and slashing memory bandwidth demands with Grouped-Query Attention (GQA) and KV quantization.
1. The Limitations of Academic Prototypes
Our project implemented KV caching, prefix sharing, and FlashAttention to understand the core algorithmic dynamics. However, in production environments serving thousands of concurrent users, our basic implementations encounter three severe hurdles:
- Memory Fragmentation from `torch.cat`: Reallocating KV tensors dynamically on every token causes GPU virtual memory fragmentation. Engines cannot predict how long a sequence will generate, resulting in reserved but unusable VRAM gaps.
- Static Tree Snapshots vs Dynamic Request Streams: Our radix tree assumed a pre-known, static batch of prompts. In a real web server, queries arrive asynchronously over time with variable prefix overlaps.
- The 70B Memory Wall: In large models like LLaMA-3-70B, storing standard multi-head KV caches for long contexts exhausts tens of gigabytes per request, severely limiting concurrent batch size.
2. PagedAttention (vLLM: Kwon et al., SOSP 2023)
Inspired by classical operating system virtual memory paging, PagedAttention solves the dynamic memory fragmentation problem:
Logical KV Cache (Tokens 0 .. 31)
┌───────────────────────┬───────────────────────┐
│ Logical Block 0 │ Logical Block 1 │
│ (Tokens 0 .. 15) │ (Tokens 16 .. 31) │
└───────────┬───────────┴───────────┬───────────┘
│ │
Block Table Map Block Table Map
▼ ▼
┌───────────────────────┐ ┌───────────────────────┐
│ Physical Block 7 │ │ Physical Block 2 │
│ (Non-contiguous) │ │ (Non-contiguous) │
└───────────────────────┘ └───────────────────────┘
GPU Physical Memory (HBM)
Key Architectural Principles
- Fixed-Size Blocks: The KV cache is divided into fixed-size physical blocks (typically 16 or 32 tokens).
- Non-Contiguous Allocation: Physical blocks do not need to be contiguous in GPU memory. A Block Table translates logical sequence offsets to physical GPU memory addresses.
- Near-Zero Waste: Eliminates external memory fragmentation entirely. Internal memory fragmentation is bounded to the very last unfilled block of a sequence (<4% memory waste).
- Copy-on-Write (CoW) Prefix Sharing: When multiple requests share a prompt prefix, their block tables point to the exact same physical blocks. Only when one sequence begins generating unique tokens does the engine allocate a private block.
3. RadixAttention (SGLang: Zheng et al., 2024)
In our project, we showed that a Radix Tree eliminates redundant prefill. RadixAttention scales this concept into a live, long-running serving daemon:
- Global Dynamic Trie: Instead of building a temporary tree per batch, the serving engine maintains a persistent global Radix Tree in GPU memory across all historical client requests.
- Automatic Prefix Matching: When a new query arrives, the engine walks the global Radix Tree to find the longest matching cached token prefix, restoring its PagedAttention blocks immediately.
- LRU Eviction Policy: When GPU VRAM reaches a high-water mark (e.g. 90% capacity), the engine does not evict entire requests. It evicts the least-recently-used leaf nodes in the radix tree, freeing memory while preserving frequently hit root system prompts.
4. Architectural Innovations: MHA vs. MQA vs. GQA
To reduce the sheer physical size of the KV cache across large models, model architectures have evolved from standard Multi-Head Attention:
Multi-Head Attention (MHA) Grouped-Query Attention (GQA) Multi-Query Attention (MQA)
Q Heads: [1] [2] [3] [4] Q Heads: [1] [2] [3] [4] Q Heads: [1] [2] [3] [4]
K Heads: [1] [2] [3] [4] K Heads: [1] [2] K Heads: [1]
V Heads: [1] [2] [3] [4] V Heads: [1] [2] V Heads: [1]
(Ratio 1:1, Max KV VRAM) (Ratio 2:1, Balanced) (Ratio 4:1, Min KV VRAM)
| Attention Architecture |
Query Heads ($H_q$) |
KV Heads ($H_{kv}$) |
KV Cache Footprint |
Used In |
| Multi-Head (MHA) |
$H$ (e.g. 32) |
$H$ (e.g. 32) |
$1.0\times$ (Baseline: 1 KV head per Q head) |
GPT-2, original Transformer |
| Multi-Query (MQA) |
$H$ (e.g. 32) |
1 |
$\mathbf{1 / H \times}$ (Massive $32\times$ reduction) |
Falcon, StarCoder |
| Grouped-Query (GQA) |
$H$ (e.g. 64) |
$G$ (e.g. 8) |
$\mathbf{1 / 8 \times}$ ($8\times$ reduction, optimal quality) |
LLaMA-2/3, Mistral 7B |
💡 Real-World Sizing Example (LLaMA-3-70B):
LLaMA-3-70B has 80 layers, 64 Query heads, 8 KV heads (GQA), and head dimension 128. For batch size 32 and context length 8,192 in FP16 precision:
$$\text{With GQA (8 KV heads)} = 2 \times 80 \times 8 \times 128 \times 8192 \times 2 \times 32 \approx \mathbf{83.9\text{ GB}}$$
$$\text{Without GQA (64 KV heads, MHA)} = 8 \times 83.9\text{ GB} \approx \mathbf{671.2\text{ GB}}$$
GQA saved nearly 600 GB of GPU memory, making high-throughput serving physically feasible!
5. KV Cache Quantization (FP8 / INT4)
Another major production axis is compressing cached tensors:
- FP8 (E4M3 / E5M2): Stores cached Keys and Values in 8-bit floating point, cutting memory by 50% and doubling serving batch capacity with negligible perplexity loss.
- INT4 / INT2: Uses per-channel or per-token outlier preservation, reducing KV memory by 75%.
6. Master Interview Summary
🎯 Interview Answer — "How do KV caching, FlashAttention, and PagedAttention fit together in a modern inference stack?"
“They tackle complementary layers of the inference bottleneck:
1. KV Caching operates at the algorithmic dataflow level: it eliminates redundant $\mathcal{O}(T^2)$ recomputation across autoregressive steps.
2. FlashAttention operates at the GPU kernel level: it tiles attention in SRAM using online softmax to bypass the $\mathcal{O}(N^2)$ memory wall.
3. PagedAttention operates at the systems memory management level: it manages KV cache pages non-contiguously like an operating system, eliminating memory fragmentation and enabling zero-copy shared prefix caching.”