Overview: This document tracks the complete end-to-end lifecycle of technical documentation in the system: from crawling raw DocBook HTML and parsing groff man pages, through recursive token-aware chunking and 384-dimensional BGE embeddings, into dual-indexed PostgreSQL 16 storage, parallel HNSW vector and GIN full-text retrieval, Reciprocal Rank Fusion (RRF), cross-encoder neural reranking, and citation-grounded LLM synthesis with evaluation metrics.
Phase 1: Ingestion
βββββββββββββββββββββββββββββββββββ βββββββββββββββββββββββββββββββββββ
β PostgreSQL 16 HTML (DocBook) β β Linux Manual Pages (groff cat) β
β Web Crawler (aiohttp) β β Subprocess (man -P cat) β
ββββββββββββββββββ¬βββββββββββββββββ ββββββββββββββββββ¬βββββββββββββββββ
β β
βΌ βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Document Extraction (extractor.py) β
β - Strip .navheader, .navfooter, .toc, script, style β
β - Extract DocBook container divs (.sect2, .sect1, .refentry) β
β - Parse man page all-caps headings (NAME, SYNOPSIS, DESCRIPTION) β
ββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββ
β
Phase 2: Chunking & Deduplication βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Recursive Token Chunker (chunker.py) β
β - Boundary hierarchy: Paragraph (\n\n+) β Sentence (?<=[.!?])\s+ β Word β
β - Token-level sizing with tiktoken (cl100k_base): target 512 tokens β
β - Sliding context overlap: 64 tokens carried forward β
β - SHA-256 content hashing for idempotency: ON CONFLICT DO NOTHING β
ββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββ
β
Phase 3: Dense Embeddings βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Embedding Generation (encoder.py) β
β - Model: BAAI/bge-small-en-v1.5 (384 dimensions, float32) β
β - Asymmetric instruction: Query prefix prepended only during search β
β - Normalized embeddings: Cosine distance <=> is equivalent to dot productβ
β - Batched inference (batch_size=32) for fast ingestion throughput β
ββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββ
β
Phase 4: Database Storage βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β PostgreSQL 16 Storage (schema.sql, connection.py) β
β - Table 'documents': id, name, source_url, doc_type β
β - Table 'chunks': chunk_id (UUID), content, token_count, content_hash β
β - Column 'content_tsv': TSVECTOR GENERATED ALWAYS AS (to_tsvector) β
β - Column 'embedding': VECTOR(384) β
β - Index 1: HNSW cosine index (m=16, ef_construction=64) β
β - Index 2: GIN inverted index on content_tsv β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
Phase 5: Candidate Retrieval βΌ (User Query Arrives)
ββββββββββββββββββββββββββ΄βββββββββββββββββββββββββ
βΌ βΌ
βββββββββββββββββββββββββββββββββ βββββββββββββββββββββββββββββ
β Channel A: Vector Search β β Channel B: Full-Text (FTS)β
β SET hnsw.ef_search = 40 β β websearch_to_tsquery(...) β
β ORDER BY embedding <=> $1 β β content_tsv @@ query β
β Top-20 candidates (Cosine) β β ts_rank_cd cover density β
βββββββββββββββββ¬ββββββββββββββββ βββββββββββββββ¬ββββββββββββββ
β β
ββββββββββββββββββββββββ¬βββββββββββββββββββββββ
βΌ
Phase 6: Rank Fusion & Rerank β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Reciprocal Rank Fusion (RRF, hybrid.py) β
β RRF_Score(chunk) = Ξ£_retriever 1 / (60 + rank_in_retriever) β
β Merges candidates without cosine vs ts_rank scale distortion β
ββββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββ
β Top-20 Candidates
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Cross-Encoder Neural Reranking (cross_encoder.py) β
β - Model: cross-encoder/ms-marco-MiniLM-L-6-v2 (22M params, 80 MB) β
β - Full joint self-attention over [CLS] Query [SEP] Passage [SEP] β
β - Prunes Top-20 candidate pool into Top-5 context chunks β
ββββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββ
β Top-5 Reranked Chunks
Phase 7: Grounded Generation βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Citation-Grounded LLM Synthesis (prompt.py, llm.py) β
β - Formats numbered context blocks: [1]...[5] with source URLs β
β - System Prompt: Answer ONLY from context; cite [Source N] on all claims β
β - Fallback guardrail: Output "insufficient context" if evidence is absentβ
β - Google GenAI SDK (Gemini 1.5 Flash / Gemma fallback) or OpenAI β
β - Regex parser extracts cited sources and builds metadata citations β
ββββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββ
β
Phase 8: Serving & Interfaces βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β FastAPI REST Server (POST /query) β Interactive Terminal CLI (cli.py) β
β Asynchronous asyncpg Pool Lifespan β REPL with /mode, /topk, ANSI Color β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
Phase 9: Evaluation Suite βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 41-Question Technical Dataset (dataset.py) β
β Recall@K, Hit@K, MRR, Citation Accuracy, LLM-Judge Hallucination Check β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
The ingestion pipeline starts in ingestion/downloader.py and ingestion/extractor.py. It ingests two very different document formats into a common normalized structure:
aiohttp from the official PostgreSQL documentation website or read from local disk mirrors.BeautifulSoup with the lxml parser and deletes all matching noise selectors:
_NOISE_SELECTORS = [
"nav", "header", "footer",
".navheader", ".navfooter", ".toc", ".table-of-contents",
"script", "style", "noscript",
".breadcrumbs", "#pgContentWrap > .nav",
]
for selector in _NOISE_SELECTORS:
for tag in soup.select(selector):
tag.decompose()
.sect2, .sect3 (granular sub-sections), falling back to .sect1, .refentry, .chapter, and .preface.
It extracts the section title from the heading element (h1..h4.title) and pairs it with the cleaned body text.
subprocess.run(["man", "-P", "cat", cmd], capture_output=True) across core utilities (e.g., ps, top, grep, awk, sed, netstat, systemctl, chmod, chown, kill, lsof)."B\x08Bo\x08ol\x08ld"). The extractor cleans these using regex: re.sub(r".\x08", "", text)._MAN_SECTION_RE = re.compile(r"^([A-Z][A-Z 0-9_-]{1,40})$", re.MULTILINE)
This cleanly splits the document into sections like NAME, SYNOPSIS, DESCRIPTION, OPTIONS, EXIT STATUS, and EXAMPLES.
Once raw sections are extracted, they are passed to RecursiveChunker in ingestion/chunker.py. Slicing technical prose arbitrarily creates fragmented sentences and broken code blocks.
The chunker implements a recursive divide-and-conquer strategy:
re.split(r"\n\n+", text).re.split(r"(?<=[.!?])\s+", text).text.split(" ").tiktokenUnlike naive chunkers that count characters (e.g., 1000 characters $\approx$ unpredictable token counts depending on whitespace and punctuation), RecursiveChunker measures exact token counts using OpenAI's fast BPE tokenizer:
self._enc = tiktoken.get_encoding("cl100k_base")
def _token_count(self, text: str) -> int:
return len(self._enc.encode(text))
chunk_size_tokens = 512. This comfortably fits within the 512-token context window of our embedding model (bge-small-en-v1.5) with zero truncation.chunk_overlap_tokens = 64. When closing a chunk, the chunker walks backwards through the merged pieces, accumulating up to 64 tokens of trailing text, and prepends them as the prefix of the next chunk. This guarantees that sentences crossing chunk boundaries remain coherent.To support incremental and resumable ingestion, each chunk calculates a SHA-256 hash of its content:
content_hash = hashlib.sha256(chunk.content.encode("utf-8")).hexdigest()
In PostgreSQL, the content_hash column is marked UNIQUE. The insertion query uses:
ON CONFLICT (content_hash) DO NOTHING. Re-running the ingestion pipeline on updated documentation skips already-processed chunks with zero overhead.
Dense semantic representations are generated by Encoder in embeddings/encoder.py:
BAAI/bge-small-en-v1.5The BGE model family was pre-trained with instruction tuning to differentiate between document indexing and retrieval queries. At query time, a task-specific prefix must be prepended:
_BGE_QUERY_PREFIX = "Represent this sentence for searching relevant passages: "
def encode_query(self, query: str) -> np.ndarray:
prefixed = _BGE_QUERY_PREFIX + query
return self._model.encode(prefixed, normalize_embeddings=True, convert_to_numpy=True).astype(np.float32)
Document passages are encoded without the prefix during ingestion. This asymmetry teaches the embedding model to bridge the semantic gap between a user question and an expository technical manual.
Embeddings are normalized to unit length ($\|v\|_2 = 1.0$) upon output. When vectors are unit-normalized, cosine distance is mathematically directly proportional to Euclidean distance and dot product:
$$\text{Cosine Distance}(u, v) = 1 - \frac{u \cdot v}{\|u\|_2 \|v\|_2} = 1 - u \cdot v$$This allows pgvector to compute cosine distance using fast SIMD dot product instructions on modern CPU and GPU hardware.
Selecting an embedding model for a production RAG system is a multi-dimensional systems trade-off between retrieval accuracy, vector memory footprint, HNSW RAM cache residency, and query latency.
| Embedding Model | Dimensions | Parameters / Disk | Vector Size (FP32) | HNSW 100K Chunks RAM | Evaluation & Trade-off Analysis |
|---|---|---|---|---|---|
BAAI/bge-small-en-v1.5 (Selected) |
384 | 33.4M (~130 MB) | 1.536 KB | ~220 MB | Optimal. Top-tier MTEB retrieval score; 512-token context window matches chunk size; small vector size keeps HNSW graph in RAM; asymmetric query prefix. |
all-MiniLM-L6-v2 |
384 | 22.7M (~90 MB) | 1.536 KB | ~220 MB | Fast and lightweight, but lacks modern contrastive hard-negative mining; delivers noticeably lower Recall@K on technical documentation and code. |
BAAI/bge-base-en-v1.5 |
768 | 109M (~440 MB) | 3.072 KB | ~440 MB | Only +1.2% higher MTEB retrieval score, but doubles HNSW RAM usage and slows dot-product vector math by 2x. Downstream cross-encoder makes this redundant. |
BAAI/bge-large-en-v1.5 |
1024 | 335M (~1.3 GB) | 4.096 KB | ~580 MB | Triples RAM consumption; 4x slower vector distance calculation. Unsuitable for fast local CPU inference. |
intfloat/e5-small-v2 |
384 | 33M (~130 MB) | 1.536 KB | ~220 MB | Strong alternative, but requires prefixes on both queries ("query: ") and documents ("passage: "). BGE stores document chunks clean without wasted prefix tokens. |
OpenAI text-embedding-3-small |
1536 (or 512 MRL) | Cloud API | 6.144 KB (or 2.048 KB) | ~880 MB (or ~290 MB) | High quality, but introduces 100β300 ms network HTTP latency, recurring API costs, vendor lock-in, and prevents offline terminal CLI usage. |
shared_buffers + OS page cache) to prevent random NVMe disk seeks during multi-layer graph navigation.
Every HNSW node stores the raw vector ($D \times 4\text{ bytes}$) plus an array of bidirectional neighbor pointers ($m \times 8\text{ bytes}$ on 64-bit systems).All relational metadata, document chunks, full-text inverted indexes, and vector embeddings reside in a single PostgreSQL 16 database initialized by database/schema.sql:
-- Enable extensions
CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
-- Documents registry table
CREATE TABLE IF NOT EXISTS documents (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL,
source_url TEXT,
doc_type TEXT CHECK (doc_type IN ('postgresql_docs', 'man_page')),
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Chunks table with dual representations
CREATE TABLE IF NOT EXISTS chunks (
chunk_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
document_id INTEGER NOT NULL REFERENCES documents (id) ON DELETE CASCADE,
document_name TEXT NOT NULL,
section TEXT,
content TEXT NOT NULL,
source_url TEXT,
token_count INTEGER,
chunk_index INTEGER,
content_hash TEXT UNIQUE,
-- Full-text search vector (automatically maintained)
content_tsv TSVECTOR GENERATED ALWAYS AS
(to_tsvector('english', content)) STORED,
-- Dense embedding (384 dimensions)
embedding VECTOR(384),
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS chunks_embedding_hnsw_idx
ON chunks
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);
m = 16: The maximum number of bidirectional connections per node in each graph layer. A value of 16 provides an optimal balance between graph navigation recall and RAM consumption.ef_construction = 64: The size of the dynamic candidate list evaluated during index construction. Higher values produce a higher-quality graph at the expense of slower indexing.vector_search.py, we execute SET hnsw.ef_search = 40 before issuing the query. This sets the size of the priority queue during traversal. Increasing ef_search from 40 to 100 improves recall on dense clusters with negligible latency increase.CREATE INDEX IF NOT EXISTS chunks_content_tsv_idx
ON chunks
USING gin (content_tsv);
content_tsv is declared as a STORED GENERATED ALWAYS column, PostgreSQL automatically parses, tokenizes, and indexes the text upon insert, requiring zero manual trigger maintenance.When a query arrives, retrieval executes across two distinct channels in retrieval/:
retrieval/vector_search.py)The user query is encoded to a 384-dimensional float vector and queried using pgvector's cosine distance operator (<=>):
SET hnsw.ef_search = 40;
SELECT
chunk_id::text,
document_name,
section,
content,
source_url,
1 - (embedding <=> $1::vector) AS score
FROM chunks
ORDER BY embedding <=> $1::vector
LIMIT 20;
<=> operator calculates cosine distance ($0.0$ for identical vectors, up to $2.0$ for opposite vectors).1 - distance converts the distance into a cosine similarity score where $1.0$ represents maximum relevance.retrieval/fts_search.py)Parallel to vector search, the raw query string is evaluated by PostgreSQL's full-text engine:
SELECT
chunk_id::text,
document_name,
section,
content,
source_url,
ts_rank_cd(content_tsv, query) AS score
FROM
chunks,
websearch_to_tsquery('english', $1) AS query
WHERE
content_tsv @@ query
ORDER BY score DESC
LIMIT 20;
websearch_to_tsquery: Parses the query using standard web-search conventions: words separated by spaces become implicit AND clauses, quoted phrases "table lock" preserve phrase order, and minus prefixes -deprecated act as NOT filters.ts_rank_cd: Uses cover-density ranking, which evaluates not just how many query terms appear in the document, but how close they appear to each other. This is significantly better for technical documentation than simple frequency counting.retrieval/hybrid.py)Now we have two ranked lists of 20 candidates each. How do we merge them?
ts_rank_cd produces unbounded floating-point scores in $[0, \infty)$ where a long chunk with dense term repetitions can achieve a score of $12.4$ while a concise chunk achieves $0.3$. Linearly summing these scores ($w_1 S_{vec} + w_2 S_{fts}$) allows FTS outliers to completely overpower vector scores.
We solve this using Reciprocal Rank Fusion (RRF) (Cormack, Clarke, & Buettcher, SIGIR 2009):
$$RRF\_Score(d) = \sum_{r \in \{\text{vector}, \text{fts}\}} \frac{1}{k + \text{rank}_r(d)}$$k = 60: Standard empirical smoothing constant. It prevents top-ranked items from dominating disproportionately over runner-ups.The top-20 RRF candidates are forwarded to Reranker in reranking/cross_encoder.py:
| Dimension | Bi-Encoder (BGE-Small) | Cross-Encoder (MS-Marco-MiniLM-L-6-v2) |
|---|---|---|
| Input Representation | Encodes query $q$ and passage $p$ separately: $E(q)$ and $E(p)$ | Concatenates query and passage together: $[CLS] \ q \ [SEP] \ p \ [SEP]$ |
| Attention Mechanism | Self-attention restricted within query tokens and within passage tokens | Full cross-attention between all query tokens and all passage tokens |
| Similarity Computation | Inner dot product / cosine distance in vector space | Single classification head on $[CLS]$ embedding output |
| Computational Cost | $\mathcal{O}(1)$ query encoding + $\mathcal{O}(\log N)$ index lookup | $\mathcal{O}(N)$ full transformer forward passes per query |
| Optimal Role in Pipeline | Fast candidate retrieval across 50,000+ chunks | High-precision reranking across the top-20 candidate pool |
The reranker constructs 20 string pairs (query, chunk["content"]) and executes a single batched forward pass on CPU:
pairs = [(query, chunk["content"]) for chunk in chunks]
scores = self._model.predict(pairs) # returns numpy array of logits
scored = sorted(zip(scores, chunks), key=lambda x: float(x[0]), reverse=True)
top_5 = scored[:5]
The top 5 chunks with the highest cross-encoder relevance scores are selected as the final context passed to the generation stage.
The top-5 chunks are passed into the generation stage (generation/prompt.py and generation/llm.py):
In prompt.py, chunks are formatted into clearly delineated, numbered context blocks:
[1] Document: postgresql_16_docs
Section: MVCC Concurrency Control
URL: https://www.postgresql.org/docs/16/mvcc-intro.html
---
PostgreSQL maintains database consistency by using a multiversion model...
[2] Document: man_pages
Section: KILL(1)
URL: https://man7.org/linux/man-pages/man1/kill.1.html
---
The command kill sends the specified signal to the specified process...
You are a precise technical documentation assistant. Your sole knowledge source
is the documentation excerpts provided below. Follow these rules strictly:
1. Answer using ONLY the information present in the provided excerpts.
2. For every factual claim, add an inline citation: [Source N] where N is the
excerpt number.
3. If multiple excerpts support a claim, cite all relevant ones: [Source 1][Source 3].
4. If the provided context is insufficient to answer the question, say explicitly:
"The provided documentation does not contain enough information to answer this question."
Do not guess or use external knowledge.
5. Be concise and technically accurate. Use bullet points or numbered lists where
appropriate for clarity.
In generation/llm.py, the client utilizes the official google.genai SDK with temperature set to $0.2$ for deterministic factual extraction. If an API quota or model error occurs, the client automatically executes an in-memory fallback cascade:
Alternatively, the client supports OpenAI's chat completions API if LLM_PROVIDER=openai.
Once the model returns its raw generated answer, extract_citations parses the text using regular expressions:
_CITATION_RE = re.compile(r"\[Source\s+(\d+)\]", re.IGNORECASE)
def extract_citations(answer: str, chunks: list[dict]) -> list[dict]:
cited_numbers = sorted({int(m) for m in _CITATION_RE.findall(answer)})
citations = []
for n in cited_numbers:
idx = n - 1
if 0 <= idx < len(chunks):
citations.append({
"source_number": n,
"document_name": chunks[idx].get("document_name"),
"section": chunks[idx].get("section"),
"source_url": chunks[idx].get("source_url"),
})
return citations
This extracts exact document names, sections, and URLs, allowing client interfaces to display interactive link chips for each citation.
api/)app.state.pool, eliminating the overhead of opening TCP connections per request.Depends(_get_conn).POST /query: Accepts {query, retrieval_mode, top_k} and returns answer, citations, chunk metadata, and execution latency.GET /health: Liveness probe validating database connectivity.GET /stats: Returns total counts of documents and chunks in the database.POST /ingest: Triggers asynchronous document ingestion in background tasks.cli.py)python cli.py starts an interactive loop with ANSI color rendering./mode <vector|hybrid|hybrid_rerank>: Hot-swaps the active retrieval pipeline on the fly./topk <1-10>: Adjusts the number of context chunks passed to the LLM./stats: Queries PostgreSQL chunk and document distribution.python cli.py "How does MVCC work?" --mode hybrid_rerank --top-k 3 for headless automation and shell scripts.In evaluation/, the system was subjected to an automated evaluation suite against a 41-question technical dataset (dataset.py) with ground-truth document and section annotations:
metrics.py)[Source N] citations that point to verified ground-truth relevant passages.| Retrieval Mode | Recall@5 | Recall@10 | Hit@5 | Hit@10 | MRR | Avg Latency |
|---|---|---|---|---|---|---|
| Vector Search (HNSW) | 0.2927 | 0.3049 | 0.3171 | 0.3171 | 0.2988 | ~18 ms |
| Hybrid Search (Vector + FTS RRF) | 0.2927 | 0.3049 | 0.3171 | 0.3171 | 0.2988 | ~26 ms |
| Hybrid + Cross-Encoder Rerank | 0.2927 | 0.3171 | 0.3171 | 0.3415 | 0.2791 | ~112 ms |