Topic 05: Benchmark Evaluation, Roofline Analysis & Hardware Trade-offs

Overview: Rigorous benchmarking methodology, empirical speedup analysis across GPU and CPU, the Roofline Model perspective explaining why prefill is compute-bound while decode is memory-bandwidth-bound, and the dual memory-vs-compute trade-offs of modern inference engines.


1. Experimental Setup & Benchmark Rigor

To ensure high scientific and systems validity across our measurements:

Component Configuration Details
Model Architecture GPT-2 Small shape: 12 layers, 12 heads, 768 embedding dim, 64 head dim, 123.7M parameters, fp32.
Hardware Platform NVIDIA GeForce RTX 3050 Laptop GPU (6 GB VRAM, sm_86, Ampere, 192 GB/s memory bandwidth) + WSL2 Linux Host.
Process Isolation Each configuration executed in an isolated worker subprocess to guarantee clean PyTorch memory allocator state.
Warmup & Timing 1 untimed warmup iteration + 5 timed repeats. Calls bracketed by `torch.cuda.synchronize()`. Reported as mean ± std.
Memory Profiling GPU: `cuda.reset_peak_memory_stats()` followed by `cuda.max_memory_allocated()`. CPU: peak RSS via `/usr/bin/time -v`.
Sampling Noise Deterministic greedy decoding (`greedy=True`) used throughout to eliminate sampling distribution variance.

2. The Roofline Model: Prefill vs. Decode Phase

In Systems for Machine Learning, the Roofline Model visualizes performance limits based on Arithmetic Intensity:

$$\text{Arithmetic Intensity} = \frac{\text{Total Floating-Point Operations (FLOPs)}}{\text{Total Memory Bytes Transferred (DRAM to SRAM)}}$$
Performance (TFLOPs)
   ▲
   │               ┌─────────────────────────────── Peak Compute Roofline
   │              /
   │             /
   │            /  ◄── Memory Bandwidth Sloped Bound (192 GB/s)
   │           /
   │          /
   │   [Decode Phase]                              [Prefill Phase]
   │   Intensity: ~1-2 FLOPs/byte                  Intensity: ~100-300 FLOPs/byte
   │   (Memory-Bandwidth Bound)                    (Compute-Bound)
   └─────────────────────────────────────────────────────────────────► Arithmetic Intensity

Why the Prefill Phase is Compute-Bound

During the prompt prefill phase, all $P$ prompt tokens are processed simultaneously in a single forward pass:

Why the Decode Phase is Memory-Bandwidth-Bound

During autoregressive decoding ($T=1$), each forward pass predicts only one single token:


3. Why GPU Speedup was Lower than CPU at Small Batch Sizes

One of the most frequent interview questions centers on our Part B results:

“Why did KV caching produce a 9.8x overall speedup on CPU, but only 3.2x on GPU at input=256, output=64?”

Device No Cache Time KV Cached Time Speedup Primary Execution Regime
CPU 9,922.3 ms 1,009.4 ms 9.83x Compute-Bound: CPU execution time tracks FLOPs directly.
GPU 1,080.8 ms 340.4 ms 3.18x Overhead-Bound: Dominated by Python launch latency.

The Architectural Explanation:

  1. At batch size 1 on an RTX 3050, the GPU has vastly more parallel compute units than the 124M model can saturate at short sequence lengths.
  2. Even without caching, the GPU finished all 64 steps in 1,080 ms (~16 ms per step).
  3. A significant portion of that 16 ms is fixed kernel launch overhead (~5–10 $\mu$s per PyTorch CUDA kernel launch $\times$ hundreds of kernel invocations per layer) plus Python interpreter overhead.
  4. KV caching eliminated redundant arithmetic, but could not eliminate the fixed kernel launch overhead per token.
  5. On CPU, there is no kernel launch overhead, and the CPU has no massive parallel compute surplus; therefore, eliminating $\mathcal{O}(T^2)$ FLOPs translated directly into a 9.8x wall-clock speedup.

4. The Dual Systems Trade-Offs: Memory vs. Compute

Our project demonstrated that systems optimization in machine learning is fundamentally an exercise in navigating trade-offs:

Trade-Off 1: Intra-Sequence Optimization (KV Caching)
Trade-Off 2: Inter-Sequence Optimization (Radix-Tree Prefix Cache)
Trade-Off 3: Kernel-Level Optimization (CUDA FlashAttention)

5. Interview Takeaway & Soundbite

🎯 Interview Answer — "How do you explain the Roofline Model difference between LLM Prefill and Decode?"
“In LLM serving, prefill and decode occupy opposite ends of the Roofline curve. Prefill forwards the entire prompt in parallel, forming large Matrix-Matrix multiplications with high arithmetic intensity, making it compute-bound and well-suited for Tensor Cores. Decode generates one token at a time, performing Matrix-Vector multiplications that must stream hundreds of megabytes of model weights and KV caches from DRAM for just a few FLOPs. Its arithmetic intensity is ~1–2 FLOPs per byte, making decode strictly memory-bandwidth bound. That is why FlashAttention accelerates prefill by saving memory traffic, while KV caching accelerates decode by eliminating redundant computations.”