Topic 03: Memory Latency Measurement & Adaptive Thresholding

Overview: How the CPU measures actual memory latency at retirement, why static latency cutoffs fail across different workloads, and how an adaptive exponential moving average dynamically separates DRAM misses from cache hits.


1. High-Level Summary

Subsystem Component Where It Runs How It Works Systems Benefit
Retirement Inspection retire_rob() Inspects completed load instructions when they reach the head of the ROB. Guarantees that total latency includes the full cache hierarchy traversal.
Latency Calculation src/ooo_cpu.cc latency = retire_time - dispatch_time Captures end-to-end cycle delay from pipeline entry to commitment.
Exponential Moving Average (EMA) avg_mem_latency Continuously updates running latency average across all retired loads. Automatically tracks phase changes in memory intensity without manual tuning.
Adaptive Threshold LATENCY_THRESHOLD Dynamic cutoff derived from running average latency. Prevents false positives on fast L1/L2 hits while reliably flagging DRAM misses.

2. Spoken Interview Answer (Simple & Direct)

"A processor needs a reliable way to decide which memory loads are actually hurting performance.

I implemented slow-load detection in the retirement stage (retire_rob()). Why retirement? Because retirement is the only point in the entire pipeline where the total, true memory latency is known—after the load has traveled through L1, L2, the Last-Level Cache, and potentially DRAM.

When a load instruction reaches the head of the Reorder Buffer, we calculate its latency as the difference between its retirement timestamp and its dispatch timestamp.

A naive implementation would use a hardcoded cutoff—for example, treating any load over 100 cycles as slow. But a static threshold fails in practice: in compute-heavy phases where most loads hit L1, 100 cycles is far too conservative. In memory-heavy phases, 100 cycles might trigger on LLC hits that don't need scheduling intervention.

To solve this, I built an adaptive threshold based on an Exponential Moving Average (EMA) of memory latency. The simulator continuously maintains a running average across all retired loads. If an individual load's latency significantly exceeds this running average, it is classified as a delinquent load.

This dynamic threshold ensures that CRISP remains sensitive during quiet phases and selective during heavy memory phases, triggering slice construction only for the loads that truly cause pipeline stalls."


3. Architecture & Mechanics

3.1 The Memory Latency Spectrum

Modern processors have a multi-tier memory hierarchy with drastically different access latencies:

Level           Typical Latency     CRISP Action
─────────────────────────────────────────────────────────────────
L1 Data Cache    ~4 to 5 cycles     Fast hit; never trigger slice
L2 Cache        ~14 cycles         Normal hit; do not prioritize
Last-Level (LLC)~40 to 50 cycles   Moderate; monitor closely
Main Memory DRAM~200+ cycles        CRITICAL DELINQUENT LOAD! Trigger slice!

3.2 How Latency is Measured at Retirement

When an instruction passes through dispatch, it receives a timestamp:

instr.dispatch_time = current_cycle;

When it finally reaches the head of the ROB and finishes retirement:

uint64_t observed_latency = current_cycle - instr.dispatch_time;

This single number captures the complete delay: queueing in the Load Queue, address generation, cache lookups, bus contention, and DRAM controller scheduling.

3.3 The Adaptive Threshold Mechanism

Rather than comparing against a fixed constant, the simulator updates a running exponential moving average:

avg_mem_latency = (alpha * observed_latency) + ((1 - alpha) * avg_mem_latency);

The threshold dynamically adjusts to three operational zones:


4. Common Interview Questions

Q1: Why can't you detect slow loads when they first issue to memory instead of at retirement?

Answer: "When a load first issues to memory from the Load-Store Queue, the CPU does not know whether it will hit in L1 (taking 4 cycles) or miss to DRAM (taking 200 cycles). The actual latency can only be observed after data returns. Measuring latency at retirement guarantees we measure the complete end-to-end memory roundtrip before classifying the instruction as delinquent."

Q2: What happens if you mark too many loads as critical (false positives)?

Answer: "If too many loads are marked critical, the priority queue in the scheduler fills up with ordinary instructions. When almost everything is marked 'critical', priority scheduling loses its effectiveness and non-critical instructions starve. The adaptive EMA threshold prevents this by ensuring that only instructions significantly slower than the workload's average latency receive priority."

Q3: What role does the smoothing factor (alpha) play in the moving average?

Answer: "The smoothing factor determines how quickly the threshold reacts to changes in program behavior. A higher alpha makes the threshold react rapidly to sudden memory spikes, while a lower alpha provides stable, smooth baseline tracking across long execution phases. We tuned alpha to maintain steady classification without oscillating on occasional isolated misses."