Topic 06: From nanoGPT to Production LLM Serving Engines

Overview: Connecting our prototype implementations in nanoGPT and CUDA FlashAttention to industrial-grade LLM inference engines (vLLM, SGLang, TensorRT-LLM): resolving memory fragmentation with PagedAttention, scaling dynamic prefix reuse with RadixAttention, and slashing memory bandwidth demands with Grouped-Query Attention (GQA) and KV quantization.


1. The Limitations of Academic Prototypes

Our project implemented KV caching, prefix sharing, and FlashAttention to understand the core algorithmic dynamics. However, in production environments serving thousands of concurrent users, our basic implementations encounter three severe hurdles:

  1. Memory Fragmentation from `torch.cat`: Reallocating KV tensors dynamically on every token causes GPU virtual memory fragmentation. Engines cannot predict how long a sequence will generate, resulting in reserved but unusable VRAM gaps.
  2. Static Tree Snapshots vs Dynamic Request Streams: Our radix tree assumed a pre-known, static batch of prompts. In a real web server, queries arrive asynchronously over time with variable prefix overlaps.
  3. The 70B Memory Wall: In large models like LLaMA-3-70B, storing standard multi-head KV caches for long contexts exhausts tens of gigabytes per request, severely limiting concurrent batch size.

2. PagedAttention (vLLM: Kwon et al., SOSP 2023)

Inspired by classical operating system virtual memory paging, PagedAttention solves the dynamic memory fragmentation problem:

Logical KV Cache (Tokens 0 .. 31)
┌───────────────────────┬───────────────────────┐
│   Logical Block 0     │   Logical Block 1     │
│   (Tokens 0 .. 15)    │   (Tokens 16 .. 31)   │
└───────────┬───────────┴───────────┬───────────┘
            │                       │
      Block Table Map         Block Table Map
            ▼                       ▼
┌───────────────────────┐ ┌───────────────────────┐
│   Physical Block 7    │ │   Physical Block 2    │
│   (Non-contiguous)    │ │   (Non-contiguous)    │
└───────────────────────┘ └───────────────────────┘
            GPU Physical Memory (HBM)

Key Architectural Principles


3. RadixAttention (SGLang: Zheng et al., 2024)

In our project, we showed that a Radix Tree eliminates redundant prefill. RadixAttention scales this concept into a live, long-running serving daemon:


4. Architectural Innovations: MHA vs. MQA vs. GQA

To reduce the sheer physical size of the KV cache across large models, model architectures have evolved from standard Multi-Head Attention:

Multi-Head Attention (MHA)       Grouped-Query Attention (GQA)       Multi-Query Attention (MQA)
  Q Heads: [1] [2] [3] [4]          Q Heads: [1] [2] [3] [4]            Q Heads: [1] [2] [3] [4]
  K Heads: [1] [2] [3] [4]          K Heads:    [1]     [2]              K Heads:        [1]
  V Heads: [1] [2] [3] [4]          V Heads:    [1]     [2]              V Heads:        [1]
  (Ratio 1:1, Max KV VRAM)          (Ratio 2:1, Balanced)               (Ratio 4:1, Min KV VRAM)
Attention Architecture Query Heads ($H_q$) KV Heads ($H_{kv}$) KV Cache Footprint Used In
Multi-Head (MHA) $H$ (e.g. 32) $H$ (e.g. 32) $1.0\times$ (Baseline: 1 KV head per Q head) GPT-2, original Transformer
Multi-Query (MQA) $H$ (e.g. 32) 1 $\mathbf{1 / H \times}$ (Massive $32\times$ reduction) Falcon, StarCoder
Grouped-Query (GQA) $H$ (e.g. 64) $G$ (e.g. 8) $\mathbf{1 / 8 \times}$ ($8\times$ reduction, optimal quality) LLaMA-2/3, Mistral 7B
💡 Real-World Sizing Example (LLaMA-3-70B):
LLaMA-3-70B has 80 layers, 64 Query heads, 8 KV heads (GQA), and head dimension 128. For batch size 32 and context length 8,192 in FP16 precision: $$\text{With GQA (8 KV heads)} = 2 \times 80 \times 8 \times 128 \times 8192 \times 2 \times 32 \approx \mathbf{83.9\text{ GB}}$$ $$\text{Without GQA (64 KV heads, MHA)} = 8 \times 83.9\text{ GB} \approx \mathbf{671.2\text{ GB}}$$ GQA saved nearly 600 GB of GPU memory, making high-throughput serving physically feasible!

5. KV Cache Quantization (FP8 / INT4)

Another major production axis is compressing cached tensors:


6. Master Interview Summary

🎯 Interview Answer — "How do KV caching, FlashAttention, and PagedAttention fit together in a modern inference stack?"
“They tackle complementary layers of the inference bottleneck: 1. KV Caching operates at the algorithmic dataflow level: it eliminates redundant $\mathcal{O}(T^2)$ recomputation across autoregressive steps. 2. FlashAttention operates at the GPU kernel level: it tiles attention in SRAM using online softmax to bypass the $\mathcal{O}(N^2)$ memory wall. 3. PagedAttention operates at the systems memory management level: it manages KV cache pages non-contiguously like an operating system, eliminating memory fragmentation and enabling zero-copy shared prefix caching.”

← Topic 05: Hardware Benchmarks Next: PyTorch Knowledge Q&A →