Overview: How the CPU measures actual memory latency at retirement, why static latency cutoffs fail across different workloads, and how an adaptive exponential moving average dynamically separates DRAM misses from cache hits.
| Subsystem Component | Where It Runs | How It Works | Systems Benefit |
|---|---|---|---|
| Retirement Inspection | retire_rob() |
Inspects completed load instructions when they reach the head of the ROB. | Guarantees that total latency includes the full cache hierarchy traversal. |
| Latency Calculation | src/ooo_cpu.cc |
latency = retire_time - dispatch_time |
Captures end-to-end cycle delay from pipeline entry to commitment. |
| Exponential Moving Average (EMA) | avg_mem_latency |
Continuously updates running latency average across all retired loads. | Automatically tracks phase changes in memory intensity without manual tuning. |
| Adaptive Threshold | LATENCY_THRESHOLD |
Dynamic cutoff derived from running average latency. | Prevents false positives on fast L1/L2 hits while reliably flagging DRAM misses. |
"A processor needs a reliable way to decide which memory loads are actually hurting performance.
I implemented slow-load detection in the retirement stage (retire_rob()). Why retirement? Because retirement is the only point in the entire pipeline where the total, true memory latency is known—after the load has traveled through L1, L2, the Last-Level Cache, and potentially DRAM.
When a load instruction reaches the head of the Reorder Buffer, we calculate its latency as the difference between its retirement timestamp and its dispatch timestamp.
A naive implementation would use a hardcoded cutoff—for example, treating any load over 100 cycles as slow. But a static threshold fails in practice: in compute-heavy phases where most loads hit L1, 100 cycles is far too conservative. In memory-heavy phases, 100 cycles might trigger on LLC hits that don't need scheduling intervention.
To solve this, I built an adaptive threshold based on an Exponential Moving Average (EMA) of memory latency. The simulator continuously maintains a running average across all retired loads. If an individual load's latency significantly exceeds this running average, it is classified as a delinquent load.
This dynamic threshold ensures that CRISP remains sensitive during quiet phases and selective during heavy memory phases, triggering slice construction only for the loads that truly cause pipeline stalls."
Modern processors have a multi-tier memory hierarchy with drastically different access latencies:
Level Typical Latency CRISP Action
─────────────────────────────────────────────────────────────────
L1 Data Cache ~4 to 5 cycles Fast hit; never trigger slice
L2 Cache ~14 cycles Normal hit; do not prioritize
Last-Level (LLC)~40 to 50 cycles Moderate; monitor closely
Main Memory DRAM~200+ cycles CRITICAL DELINQUENT LOAD! Trigger slice!
When an instruction passes through dispatch, it receives a timestamp:
instr.dispatch_time = current_cycle;
When it finally reaches the head of the ROB and finishes retirement:
uint64_t observed_latency = current_cycle - instr.dispatch_time;
This single number captures the complete delay: queueing in the Load Queue, address generation, cache lookups, bus contention, and DRAM controller scheduling.
Rather than comparing against a fixed constant, the simulator updates a running exponential moving average:
avg_mem_latency = (alpha * observed_latency) + ((1 - alpha) * avg_mem_latency);
The threshold dynamically adjusts to three operational zones:
Answer: "When a load first issues to memory from the Load-Store Queue, the CPU does not know whether it will hit in L1 (taking 4 cycles) or miss to DRAM (taking 200 cycles). The actual latency can only be observed after data returns. Measuring latency at retirement guarantees we measure the complete end-to-end memory roundtrip before classifying the instruction as delinquent."
Answer: "If too many loads are marked critical, the priority queue in the scheduler fills up with ordinary instructions. When almost everything is marked 'critical', priority scheduling loses its effectiveness and non-critical instructions starve. The adaptive EMA threshold prevents this by ensuring that only instructions significantly slower than the workload's average latency receive priority."
Answer: "The smoothing factor determines how quickly the threshold reacts to changes in program behavior. A higher alpha makes the threshold react rapidly to sudden memory spikes, while a lower alpha provides stable, smooth baseline tracking across long execution phases. We tuned alpha to maintain steady classification without oscillating on occasional isolated misses."