Topic 05: Benchmark Evaluation, Roofline Analysis & Hardware Trade-offs
Overview: Rigorous benchmarking methodology, empirical speedup analysis across GPU and CPU, the Roofline Model perspective explaining why prefill is compute-bound while decode is memory-bandwidth-bound, and the dual memory-vs-compute trade-offs of modern inference engines.
1. Experimental Setup & Benchmark Rigor
To ensure high scientific and systems validity across our measurements:
| Component |
Configuration Details |
| Model Architecture |
GPT-2 Small shape: 12 layers, 12 heads, 768 embedding dim, 64 head dim, 123.7M parameters, fp32. |
| Hardware Platform |
NVIDIA GeForce RTX 3050 Laptop GPU (6 GB VRAM, sm_86, Ampere, 192 GB/s memory bandwidth) + WSL2 Linux Host. |
| Process Isolation |
Each configuration executed in an isolated worker subprocess to guarantee clean PyTorch memory allocator state. |
| Warmup & Timing |
1 untimed warmup iteration + 5 timed repeats. Calls bracketed by `torch.cuda.synchronize()`. Reported as mean ± std. |
| Memory Profiling |
GPU: `cuda.reset_peak_memory_stats()` followed by `cuda.max_memory_allocated()`. CPU: peak RSS via `/usr/bin/time -v`. |
| Sampling Noise |
Deterministic greedy decoding (`greedy=True`) used throughout to eliminate sampling distribution variance. |
2. The Roofline Model: Prefill vs. Decode Phase
In Systems for Machine Learning, the Roofline Model visualizes performance limits based on Arithmetic Intensity:
$$\text{Arithmetic Intensity} = \frac{\text{Total Floating-Point Operations (FLOPs)}}{\text{Total Memory Bytes Transferred (DRAM to SRAM)}}$$
Performance (TFLOPs)
▲
│ ┌─────────────────────────────── Peak Compute Roofline
│ /
│ /
│ / ◄── Memory Bandwidth Sloped Bound (192 GB/s)
│ /
│ /
│ [Decode Phase] [Prefill Phase]
│ Intensity: ~1-2 FLOPs/byte Intensity: ~100-300 FLOPs/byte
│ (Memory-Bandwidth Bound) (Compute-Bound)
└─────────────────────────────────────────────────────────────────► Arithmetic Intensity
Why the Prefill Phase is Compute-Bound
During the prompt prefill phase, all $P$ prompt tokens are processed simultaneously in a single forward pass:
- Matrix multiplications ($Q K^T$ and $\text{Attn} \cdot V$) operate on shapes $(P, d) \times (d, P)$.
- FLOPs scale quadratically with prompt length $\mathcal{O}(P^2 \cdot d)$, while memory traffic scales linearly with weights and inputs $\mathcal{O}(P \cdot d + d^2)$.
- Arithmetic Intensity is proportional to $P$. For large prompts, operational intensity easily crosses the Roofline knee point. The GPU Tensor Cores are fully utilized.
Why the Decode Phase is Memory-Bandwidth-Bound
During autoregressive decoding ($T=1$), each forward pass predicts only one single token:
- Matrix-vector multiplications (GEMV) of shape $(1, d) \times (d, d)$.
- Computes only $\mathcal{O}(d^2)$ FLOPs per layer.
- The Critical Cost: To compute those few FLOPs, the GPU must fetch all 495 MB of model weights and all historical KV cache tensors from global HBM over the memory bus!
- Arithmetic Intensity drops to just ~1 to 2 FLOPs per byte.
- The GPU compute cores sit idle ~95% of the time, waiting on memory transfers from DRAM. The generation speed is strictly determined by memory bandwidth: $\text{Tokens/sec} \approx \frac{\text{Memory Bandwidth}}{\text{Model Weight Bytes} + \text{KV Cache Bytes}}$.
3. Why GPU Speedup was Lower than CPU at Small Batch Sizes
One of the most frequent interview questions centers on our Part B results:
“Why did KV caching produce a 9.8x overall speedup on CPU, but only 3.2x on GPU at input=256, output=64?”
| Device |
No Cache Time |
KV Cached Time |
Speedup |
Primary Execution Regime |
| CPU |
9,922.3 ms |
1,009.4 ms |
9.83x |
Compute-Bound: CPU execution time tracks FLOPs directly. |
| GPU |
1,080.8 ms |
340.4 ms |
3.18x |
Overhead-Bound: Dominated by Python launch latency. |
The Architectural Explanation:
- At batch size 1 on an RTX 3050, the GPU has vastly more parallel compute units than the 124M model can saturate at short sequence lengths.
- Even without caching, the GPU finished all 64 steps in 1,080 ms (~16 ms per step).
- A significant portion of that 16 ms is fixed kernel launch overhead (~5–10 $\mu$s per PyTorch CUDA kernel launch $\times$ hundreds of kernel invocations per layer) plus Python interpreter overhead.
- KV caching eliminated redundant arithmetic, but could not eliminate the fixed kernel launch overhead per token.
- On CPU, there is no kernel launch overhead, and the CPU has no massive parallel compute surplus; therefore, eliminating $\mathcal{O}(T^2)$ FLOPs translated directly into a 9.8x wall-clock speedup.
4. The Dual Systems Trade-Offs: Memory vs. Compute
Our project demonstrated that systems optimization in machine learning is fundamentally an exercise in navigating trade-offs:
Trade-Off 1: Intra-Sequence Optimization (KV Caching)
- Saved: Compute FLOPs ($\mathcal{O}(T^2) \to \mathcal{O}(T)$). Decouples per-step decode latency from historical sequence length (up to 29.1x decode acceleration).
- Cost: Memory capacity ($2 \times L \times H \times d \times T \times \text{bytes}$). High transient memory allocation churn if using naive concatenation.
Trade-Off 2: Inter-Sequence Optimization (Radix-Tree Prefix Cache)
- Saved: Redundant prompt prefill FLOPs across batched queries (1.25x GPU / 2.03x CPU speedup at prefix=256).
- Cost: Multiple active branch KV cache snapshots held simultaneously in VRAM (peak memory increases from flat 496 MB to 544 MB).
Trade-Off 3: Kernel-Level Optimization (CUDA FlashAttention)
- Saved: Eliminates global memory traffic and reduces auxiliary memory from $\mathcal{O}(N^2)$ to $\mathcal{O}(N)$. Prevents OOM at 4K context.
- Cost: Requires higher arithmetic intensity and on-chip SRAM synchronization (`__syncthreads()`).
5. Interview Takeaway & Soundbite
🎯 Interview Answer — "How do you explain the Roofline Model difference between LLM Prefill and Decode?"
“In LLM serving, prefill and decode occupy opposite ends of the Roofline curve. Prefill forwards the entire prompt in parallel, forming large Matrix-Matrix multiplications with high arithmetic intensity, making it compute-bound and well-suited for Tensor Cores. Decode generates one token at a time, performing Matrix-Vector multiplications that must stream hundreds of megabytes of model weights and KV caches from DRAM for just a few FLOPs. Its arithmetic intensity is ~1–2 FLOPs per byte, making decode strictly memory-bandwidth bound. That is why FlashAttention accelerates prefill by saving memory traffic, while KV caching accelerates decode by eliminating redundant computations.”