Topic 05: Traceback Frame Extraction & Runtime Error Diagnosis

Target Duration: 2–4 minutes (~300–450 spoken words)
Focus: Pointwise verbal delivery covering regex stack frame extraction, interval containment matching, the backwards `.endswith()` path resolution bug, hybrid frame-plus-semantic context gathering, and root-cause synthesis.

🎙️ Pointwise Spoken Speech (Word-for-Word Delivery)

• Opening & Scope:
  "In this subsystem (sherlock/router.py:36 (diagnose)), I designed an automated error diagnosis engine that parses raw execution stack traces, maps crashed line numbers back to their exact AST chunks, and orchestrates root-cause diagnosis through local LLM synthesis."

• Step 1: The Problem with Naive Error RAG:
  "Most RAG coding assistants treat pasted error logs as plain text, passing the entire traceback into a vector embedding model. This fails because compiler and interpreter stack traces are dense with boilerplate, absolute filesystem paths, and library frames. Vector search alone rarely surfaces the actual line of application code where the exception occurred."

• Step 2: Dual Regex Stack Frame Parsing:
  "In parse_frames() (router.py:21), I implemented a multi-stage regex extractor:

  • First, it applies PY_FRAME_RE (File "([^"]+)", line (\d+)) to extract Python stack traces.
  • If no Python frames match, it falls back to GENERIC_FRAME_RE (([^\s():]+\.\w+):(\d+)) to parse GCC compiler errors, Node.js traces, and Go panics.
  • Extracted frames are deduplicated in order, isolating distinct (file, line) coordinates."

• Step 3: Interval Containment Line Matching:
  "Once the executing (file, line) tuples are parsed, Sherlock locates the exact enclosing function in router.py:45. Because each indexed chunk stores physical boundaries (start_line and end_line), the router evaluates interval containment: r["start_line"] <= line <= r["end_line"]. This pulls the exact function that crashed, providing complete syntactic context rather than just a detached single line."

• Step 4: The Backwards Path Suffix Bug:
  "A critical bug arose during early testing: tracebacks use absolute paths like /Users/.../app.py, while the index stores relative paths like app.py. The code originally ran r["file"].endswith(file)—checking if the short path ended with the long path—which failed 100% of the time! I diagnosed and resolved this by reversing the evaluation to file.endswith(r["file"])."

• Step 5: Hybrid Context & Root-Cause Synthesis:
  "Finally, in router.py:49, Sherlock extracts the terminal exception line (e.g. ZeroDivisionError: division by zero) and executes a supplementary vector search for semantically related code. Combining the executing frame chunks with semantic neighbor chunks, it prompts Ollama with a strict directive in SYSTEM_PROMPT: identify the root cause first, then suggest a concrete fix."

📋 Step-by-Step Summary (What, How & Why)

🔍 Under-the-Hood Deep Dive: Frame Matching Engine

# Exact stack frame resolution in router.py
def diagnose(repo_root: Path, traceback_text: str, k: int = 5) -> dict:
    frames = parse_frames(traceback_text)
    frame_chunks = []
    
    # 1. Deterministic stack frame interval match
    for file, line in frames:
        table = indexer.open_table(repo_root)
        rows = table.search().to_list()
        matches = [
            r for r in rows
            if file.endswith(r["file"]) and r["start_line"] <= line <= r["end_line"]
        ]
        frame_chunks.extend(matches)

    # 2. Semantic vector search on exception description
    error_line = traceback_text.strip().splitlines()[-1] if traceback_text.strip() else ""
    semantic = indexer.semantic_search(repo_root, error_line, k=k) if error_line else []

    return {
        "type": "diagnose",
        "frames": frames,
        "frame_chunks": frame_chunks,
        "semantic_chunks": semantic,
    }

⚠️ Debugging Stories: The Backwards Path Suffix Trap

The Failure

In early testing of sherlock diagnose, feeding valid Python tracebacks produced zero matched stack frames. The CLI repeatedly reported:

-- matched stack frames --
(empty)
-- semantically related --
...

Diagnosis & Root Cause

Inspecting the frame matching logic revealed:

# BUGGY ORIGINAL LINE:
r["file"].endswith(file.lstrip("./"))

• r["file"] was the indexed relative path: "app.py"
• file was the traceback path extracted by regex: "/Users/angshuman/git/test_project/app.py"

The Resolution

Reversed the comparison argument in router.py:45 to:

file.endswith(r["file"])

💡 Tough Interview Questions & Detailed Answers

Q1: "Why combine stack frame matching with semantic search on the exception line?"

Answer: The stack trace identifies where the program crashed, but often the root cause originated elsewhere (e.g. an invalid config loaded at startup, or a caller passing unvalidated None). Frame interval matching guarantees you retrieve the exact crashed function, while semantic search on the exception message (e.g., "KeyError: 'timeout'") retrieves configuration loaders, dictionary definitions, or upstream callers that handle timeouts.

Q2: "What happens if a traceback refers to third-party library files in site-packages?"

Answer: Because .venv, site-packages, and Python standard libraries are excluded from the repository index by SKIP_DIRS, file.endswith(r["file"]) simply yields no match for library frames. Only the stack frames corresponding to first-party application code are matched, filtering out external framework noise and focusing the LLM's context window entirely on the user's code.