Topic 03: Content-Hash Incremental Indexing & Manifest Diffing

Target Duration: 2–4 minutes (~300–450 spoken words)
Focus: Pointwise verbal delivery covering SHA-1 content hashing, manifest diffing algorithms, SQL OR-chained row invalidation, and real-world benchmark metrics (1.49s no-op vs multi-minute full rebuild).

🎙️ Pointwise Spoken Speech (Word-for-Word Delivery)

• Opening & Scope:
  "In this subsystem (indexer.py:112 (build_index)), I engineered a content-hash incremental re-indexing pipeline that reduces re-indexing times on large codebases from minutes to sub-second runs by caching per-file SHA-1 signatures."

• Step 1: The Full-Rebuild Bottleneck:
  "In the naive implementation, re-indexing required re-parsing and re-embedding every single file in the repository on every invocation. When testing against a production repository containing 495 files and 5,231 chunks, a full rebuild took several minutes and exceeded execution timeouts. For a CLI tool to be usable during active development, re-indexing must be near-instantaneous."

• Step 2: The SHA-1 Manifest Architecture:
  "To track file state across runs, I introduced a lightweight manifest at .sherlock/manifest.json (indexer.py:21). Stored as a simple JSON dictionary mapping relative file paths to their 40-character hex SHA-1 digests ({rel_path: sha1_hex}), this manifest persists the exact cryptographic state of every indexed file."

• Step 3: Fast Diffing Algorithm (Changed vs Removed):
  "When build_index() runs, it loads the prior manifest via _load_manifest().

  • It walks discovered files, computes each file's SHA-1 digest, and checks: if old_manifest.get(rel) != h: changed_paths.append(path).
  • It identifies deleted files using set difference: removed_rels = set(old_manifest) - set(new_manifest).
  • If a file's hash is unchanged, it bypasses Tree-sitter AST parsing and FastEmbed tensor inference entirely!"

• Step 4: Atomic Row Invalidation via SQL OR Chains:
  "For changed or deleted files, old chunks in LanceDB must be purged to prevent duplicate or phantom results. In _sql_or() (indexer.py:80), I built a query generator that escapes single quotes and constructs an atomic SQL predicate: file = 'path1' OR file = 'path2'. In indexer.py:134, table.delete(to_delete) purges all stale rows in a single batch before newly embedded chunks are appended via table.add(rows)."

• Step 5: Verified Benchmark Results:
  "I validated this pipeline on the 495-file Conductor repository:

  • A no-op re-index (no files modified) took only 1.49 seconds.
  • Appending a single line to a container shell script was detected and re-indexed in just 0.984 seconds, re-embedding only that single modified file."

📋 Step-by-Step Summary (What, How & Why)

📊 Performance Benchmarks & Empirical Results

The incremental indexing engine was benchmarked against the real Conductor codebase (495 files, 5,231 chunks, including C, Python, Shell, and JSON):

🔍 Under-the-Hood Deep Dive: The Diff Algorithm

The incremental update logic in indexer.py guarantees transactional consistency:

# Set difference for removed files
removed_rels = set(old_manifest) - set(new_manifest)
changed_rels = {str(p.relative_to(repo_root)) for p in changed_paths}

# Atomic row deletion via OR predicate
to_delete = changed_rels | removed_rels
if to_delete:
    table.delete(_sql_or("file", to_delete))

# Re-chunk and embed only changed files
new_chunks = []
for path in changed_paths:
    new_chunks.extend(chunk_file(path, repo_root))
rows = _rows(new_chunks)
if rows:
    table.add(rows)

# Atomically update disk manifest
_save_manifest(repo_root, new_manifest)

💡 Tough Interview Questions & Detailed Answers

Q1: "Why use SHA-1 content hashing rather than file modification time (mtime)?"

Answer: mtime timestamps are notoriously unreliable in software development workflows. Running git checkout, switching git branches, running build scripts, or touching files updates mtime without altering file contents, which would trigger expensive, unnecessary re-embeddings across hundreds of files. Cryptographic SHA-1 hashing ensures that a file is only re-embedded when its actual bytes change, guaranteeing 100% precision.

Q2: "What is the trade-off of file-level vs chunk-level incremental diffing?"

Answer: Sherlock uses file-level diffing: modifying one line in a 50-function file causes all 50 functions in that file to be re-embedded. A chunk-level diffing engine would hash individual AST nodes and only re-embed the mutated function. While chunk-level diffing would save embedding cycles on massive files, it adds significant complexity (tracking AST node hashes, handling line shifts for downstream functions in the same file). For codebases up to 1,000 files, file-level diffing provides sub-second re-indexing with minimal architectural complexity.