PyTorch Knowledge & KV Caching Internals Guide

Overview: Essential technical interview questions and deep-dive architectural answers on PyTorch internals and KV caching mechanics in nanoGPT. Covers 4D tensor manipulation, view vs transpose contiguousness, caching lifecycle, causal masking across execution phases, device placement, Autograd inference optimization, and memory allocator pitfalls.


Part 1: KV Caching in nanoGPT (Deep Knowledge)

1.1 Tensor Shapes: `[batch, heads, sequence_length, head_dim]`

In multi-head attention, PyTorch organizes activation tensors as 4D shapes:

Shape: (B, nh, T, hs)
  B  = Batch Size (number of parallel sequences)
  nh = Number of Attention Heads (e.g., 12 in GPT-2 Small)
  T  = Sequence Length (number of tokens in the current slice)
  hs = Head Dimension (features per head, e.g., 64)
  Total Embedding Dimension C = nh * hs = 12 * 64 = 768
🎯 Why are Heads placed before Sequence Length: `(B, nh, T, hs)` instead of `(B, T, nh, hs)`?
PyTorch's batched matrix multiplication (`torch.matmul` or `@`) operates on the last two dimensions of a tensor, treating all preceding leading dimensions as batch dimensions. By structuring tensors as `(B, nh, T_q, hs)` and `(B, nh, hs, T_k)`, PyTorch executes $B \times nh$ independent 2D matrix multiplications $(T_q \times hs) \times (hs \times T_k) \to (T_q \times T_k)$ across all attention heads in parallel with a single fused kernel call!

1.2 Fundamental PyTorch Ops: `view()`, `transpose()`, `matmul()`, `cat()`, and `softmax()`

PyTorch Operation What It Does in Self-Attention Memory & Systems Implications
`view(*shape)` Splits the flat hidden dimension into heads: `x.view(B, T, nh, hs)`. Zero-copy view sharing the same underlying data buffer. Requires the tensor to be contiguous in memory; otherwise throws a `RuntimeError`.
`transpose(dim0, dim1)` Swaps dimensions 1 and 2 to position heads before time: `x.transpose(1, 2)`. Does not reorder bytes in memory! Modifies only tensor metadata (strides). The resulting tensor is non-contiguous, requiring `.contiguous()` before calling `.view()`.
`matmul` (`@`) Computes $Q @ K^T \to \text{Scores}$ and $\text{Attn} @ V \to \text{Output}$. Dispatches to optimized cuBLAS / CUTLASS GEMM kernels on GPU. Broadcasts across batch and head dimensions.
`cat(tensors, dim)` Appends new token Key/Value vectors to cache: `torch.cat((cache_k, k), dim=2)`. Always allocates a new memory buffer and copies both tensors. Causes transient memory doubling and allocation overhead during decoding.
`softmax(x, dim)` Normalizes attention scores along key columns: `F.softmax(att, dim=-1)`. Subtracts row maximum ($\max_j x_j$) internally before exponentiating to prevent numerical overflow ($\exp(x) \to \infty$).

1.3 How K/V Tensors are Cached and Updated in `model.py`

In `CausalSelfAttention`, Key and Value tensors are managed across the generation lifecycle:

class CausalSelfAttention(nn.Module):
    def __init__(self, config):
        super().__init__()
        # ... standard projections ...
        self.use_cache = False
        self.cache_k = None
        self.cache_v = None

    def reset_cache(self):
        self.cache_k = None
        self.cache_v = None

    def forward(self, x):
        B, T, C = x.size()
        # 1. Project Q, K, V for input tokens (B, T, C) -> (B, nh, T, hs)
        q, k, v = self.c_attn(x).split(self.n_embd, dim=2)
        q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)

        # 2. KV Cache Update
        if self.use_cache:
            if self.cache_k is not None:
                # Decoding phase: concatenate newest token (T=1) along sequence dim (dim=2)
                k = torch.cat((self.cache_k, k), dim=2)
                v = torch.cat((self.cache_v, v), dim=2)
            # Store updated context in cache
            self.cache_k = k
            self.cache_v = v

        T_ctx = k.size(2)  # Total context length including past tokens
        # 3. Scaled Dot-Product Attention over T_ctx keys
        # ...

1.4 Causal Masking: Prefill vs. Single-Token Decode vs. Chunked Prefill

Prefill Phase (T = T_ctx = 4):            Decode Phase (T = 1, T_ctx = 4):
Query tokens: 0, 1, 2, 3                   Query token: 3 (newest token)

Keys:  0   1   2   3                       Keys:  0   1   2   3
Q0:  [ 1   0   0   0 ] (sees 0)            Q3:  [ 1   1   1   1 ] (sees 0..3)
Q1:  [ 1   1   0   0 ] (sees 0..1)
Q2:  [ 1   1   1   0 ] (sees 0..2)         * No mask needed! Query at index T_ctx-1
Q3:  [ 1   1   1   1 ] (sees 0..3)           attends to all past keys causally.
* Square Lower-Triangular Mask

1.5 Device Placement, CUDA Streams & Tensor Allocation Lifecycle


Part 2: PyTorch Fundamentals (Core Basics)

2.1 `torch.Tensor` Anatomy, Strides & Memory Layout

A `torch.Tensor` consists of a Storage buffer (a 1D block of contiguous memory containing raw data bytes) plus a View / Layout (metadata defining dimensions, data type, and strides):

$$\text{Memory Offset}(\text{indices}) = \sum_{d=0}^{\text{dim}-1} \text{index}_d \times \text{stride}_d$$

2.2 `nn.Module` Architecture & The `forward()` Pipeline


2.3 `requires_grad`, `torch.no_grad()`, and `torch.inference_mode()`

Mechanism Autograd Graph Built? Tensor Version Tracking? Primary Use Case
`requires_grad=True` Yes (records operation history) Yes Model training (backward pass / backprop).
`with torch.no_grad():` No (saves VRAM, no activations saved) Yes (tracks in-place mutations) Standard evaluation / inference.
`with torch.inference_mode():` No No (strictest & fastest mode) High-performance production inference serving.
πŸ’‘ Why does `torch.inference_mode()` outperform `torch.no_grad()`?
In `torch.no_grad()`, PyTorch still tracks tensor version counters to detect illegal in-place mutations. `torch.inference_mode()` disables tensor version tracking entirely and marks tensors as non-differentiable views. This reduces C++ runtime overhead and allows PyTorch compilers (like `torch.compile`) to generate more aggressive kernel fusions.

2.4 CPU vs. CUDA Tensors: Architectural Differences

Dimension CPU Tensors CUDA Tensors
Physical Location Host DDR4 / DDR5 RAM GPU High Bandwidth Memory (GDDR6 / HBM)
Bandwidth ~50 to 100 GB/s 192 GB/s (RTX 3050) to 3.35 TB/s (H100)
Parallelism 4 to 64 CPU cores (SIMD / AVX-512) Thousands of CUDA cores & Tensor Cores
Execution Dispatch Synchronous execution per thread Asynchronous launch onto CUDA Streams
Copy Overhead Zero-copy between CPU processes Requires PCIe bus transfers (~16–32 GB/s bottleneck)

2.5 Why Unnecessary Tensor Copies & Allocations Hurt Performance

In high-throughput LLM decoding, unnecessary tensor copies degrade systems performance across four axes:

  1. Memory Allocator Stalls: Dynamic allocations inside generation loops (`torch.cat`, `.clone()`) query PyTorch's caching allocator. When no cached block of that size exists, the allocator triggers internal mutex locks or calls `cudaMalloc`, blocking the CPU thread.
  2. Memory Bandwidth Contention: The decode phase has an arithmetic intensity of only ~1–2 FLOPs/byte and is strictly memory-bandwidth bound. Every redundant byte copied over the memory bus directly robs bandwidth from fetching model weights and KV caches!
  3. Heap Fragmentation & False OOM: Repeatedly creating and freeing tensors of varying sizes fragments GPU virtual memory. Over time, the largest contiguous free chunk shrinks, triggering a `CUDA OutOfMemory` crash even when 40% of total VRAM is technically free.
  4. Pipeline Stalls from Device Transfers: Calling `.cpu()` or `.item()` forces the CPU to wait for all queued GPU work to complete, draining the GPU execution queue and idling Tensor Cores.

Part 3: Interview Master Cheat Sheet

🎯 Interview Soundbite β€” "How do you explain the PyTorch memory implications of KV Caching?"
β€œIn PyTorch, KV caching trades memory capacity to eliminate $\mathcal{O}(T^2)$ compute. However, a naive implementation using `torch.cat` on every token causes an allocate-before-free memory churn where old and new tensors temporarily co-exist, doubling transient memory. Furthermore, tensors must be reshaped into `(B, nh, T, hs)` so that PyTorch can dispatch batched matrix multiplications across all heads in parallel. To productionize this, engines like vLLM replace dynamic `torch.cat` with pre-allocated non-contiguous memory blocks via PagedAttention.”

← Topic 06: Production Serving Attention Hub Index β†’