4–5 Minute Detailed Architecture: RAG Technical Documentation Assistant
Target Duration: ~4 to 5 minutes (~800–950 spoken words)
Tone: Systems architect, deeply technical, precise on algorithmic and database trade-offs.
Structure: 5 Architectural Stages covering Ingestion/Chunking, Single-Store PostgreSQL/pgvector Indexing, Hybrid RRF Retrieval, Cross-Encoder Reranking, and Citation-Grounded Generation/Evaluation.
🎙️ Complete Architectural Presentation Script
“To explain the architecture and systems engineering behind this project, I break it down into five key stages: the ingestion and token-aware chunking pipeline, unified dual-index PostgreSQL database storage, hybrid retrieval with Reciprocal Rank Fusion, cross-encoder neural reranking, and citation-grounded LLM synthesis with benchmark evaluation.
Stage 1: Ingestion Pipeline & Recursive Token-Aware Chunking
The first challenge was preparing a heterogeneous technical corpus comprising official PostgreSQL 16 HTML documentation and Linux system manual pages.
In standard RAG tutorials, documents are sliced into naive character-based or fixed-line windows. On technical manuals, this causes disastrous splits: code snippets get sliced mid-statement, command flags become separated from their explanations, and table headers are orphaned. I built a multi-stage parser:
- For PostgreSQL DocBook HTML, I used BeautifulSoup to strip navigation headers, footers, breadcrumbs, and tables of contents, targeting specific DocBook structural containers like
.sect2, .sect1, and .refentry.
- For Linux man pages, I stripped groff backspace formatting codes and used regular expressions to split on all-caps section headings such as
NAME, SYNOPSIS, and DESCRIPTION.
- For chunking, I designed a recursive text chunker that prioritizes natural syntactic boundaries: paragraph breaks first, then sentence boundaries, and finally words as a last resort.
- Crucially, sizing is enforced using
tiktoken with the cl100k_base tokenizer, targeting 512 tokens per chunk with a 64-token overlap window carried forward to preserve cross-chunk context.
- Finally, every chunk is hashed using SHA-256 to guarantee idempotent database insertion: re-running ingestion skips existing chunks without duplicate writes.”
(Hook: Sets up deep-dive into chunk boundary strategies and embedding batching)
Stage 2: Unified Dual-Index Database Engine (PostgreSQL + pgvector)
“A pivotal design decision was avoiding external dedicated vector databases like Pinecone, Qdrant, or Milvus in favor of a single-store architecture in PostgreSQL 16 using pgvector.
In production systems, running a separate vector database alongside a relational database introduces severe distributed systems overhead: dual-write transaction failures, replication lag between metadata and vectors, and complex network topologies. PostgreSQL satisfies all our requirements in one ACID-compliant engine:
- I designed a
chunks table storing raw text, document metadata, an automatically maintained full-text search vector content_tsv (generated via to_tsvector('english', content)), and a 384-dimensional VECTOR(384) column.
- For vector indexing, I built an HNSW (Hierarchical Navigable Small World) index using cosine distance (
vector_cosine_ops). I configured it with $m=16$ bidirectional links per node and an index-build search depth of $ef_{construction}=64$.
- To balance latency and recall at query time, I dynamically set
SET hnsw.ef_search = 40 per database transaction, allowing fine-tuning without rebuilding the index.
- Alongside HNSW, I created a GIN inverted index over the
content_tsv column for instant keyword lookups.”
(Hook: Sets up questions on HNSW graph memory footprint vs. IVFFlat list probing)
Stage 3: Hybrid Retrieval & Reciprocal Rank Fusion (RRF)
“During retrieval, relying solely on dense vector embeddings creates significant blind spots for technical documentation. Vector models capture semantic concepts well—like asking 'how does concurrency work?'—but they frequently fail on exact keywords, command-line flags (such as grep -E versus grep -F), or specific configuration variables like max_connections.
To overcome this, I built a hybrid retrieval engine that queries both indexes in parallel:
- The user query is encoded with
BAAI/bge-small-en-v1.5 using its asymmetric query instruction prefix to retrieve the top-20 candidates via pgvector's cosine distance operator (<=>).
- Simultaneously, PostgreSQL's
websearch_to_tsquery executes an exact lexical query over the GIN index, ranked by ts_rank_cd cover-density ranking, retrieving another 20 candidates.
- Merging these lists presents a classic systems problem: cosine similarity is bounded between 0 and 1, whereas
ts_rank_cd produces unbounded scores based on term proximity. A linear combination like $\alpha S_{vec} + \beta S_{fts}$ is notoriously fragile.
- Instead, I implemented Reciprocal Rank Fusion (RRF) with smoothing constant $k=60$:
$$RRF(d) = \sum_{r \in \{\text{vector}, \text{fts}\}} \frac{1}{60 + \text{rank}_r(d)}$$
Because RRF operates strictly on ordinal ranks, it is completely immune to score scale mismatch. Documents appearing near the top of both lists receive compounded boosts, while unique hits from either retriever are preserved.”
(Hook: Sets up discussion on score normalization vs. rank aggregation)
Stage 4: Cross-Encoder Neural Reranking
“Even after RRF fusion, bi-encoder retrieval has an intrinsic limitation: bi-encoders map query and document independently into fixed-dimensional vectors, meaning all term interaction is compressed into a simple dot product.
To maximize precision before passing context to the LLM, I integrated a Cross-Encoder reranking model (cross-encoder/ms-marco-MiniLM-L-6-v2):
- The cross-encoder concatenates the query and candidate chunk into a single sequence:
[CLS] Query [SEP] Passage [SEP], feeding them into 6 transformer layers.
- This allows full self-attention across every query token and passage token simultaneously, capturing subtle lexical negations and modifier relationships that bi-encoders miss.
- Because cross-encoder inference is $\mathcal{O}(N)$ in the number of candidates, running it over the full database is computationally prohibitive. In our pipeline, we retrieve 20 candidates in ~25 ms via database indexes, and then evaluate the cross-encoder strictly over those 20 candidates in ~80 ms on CPU, selecting the top-5 highest-scoring chunks.”
(Hook: Sets up deep-dive into computational complexity and two-stage search pipelines)
Stage 5: Citation-Grounded LLM Generation, Serving & Empirical IR Evaluation
“The final stage is grounded answer synthesis, serving, and quantitative evaluation:
- Context Injection & Grounding Contract: I format the top-5 reranked chunks into numbered context blocks containing the document name, section title, official source URL, and text. The system prompt imposes a strict contract: the LLM must answer exclusively from the provided excerpts, must append an inline citation
[Source N] to every factual statement, and must explicitly output 'insufficient context' if the documentation does not contain the answer, eliminating hallucinations. A regex layer parses these citations and validates them against the provided chunks.
- Asynchronous Serving: For production serving, I implemented an async FastAPI backend utilizing an
asyncpg connection pool with custom pgvector type codecs, alongside an interactive terminal CLI (cli.py) supporting REPL slash commands.
- Rigorous Information Retrieval (IR) Evaluation: To rigorously measure retrieval performance rather than relying on qualitative impressions, I built an automated evaluation harness across a 41-question technical benchmark with annotated ground-truth documentation sections. I track three core IR metrics:
- Recall@K (Coverage): Measures whether all relevant documentation chunks appear among the top $K$ retrieved results. In our benchmarks, Recall@10 increased from 30.49% to 31.71% when adding the cross-encoder.
- Hit@K (Success Rate): Measures whether at least one relevant chunk appears among the top $K$ results (binary recall). Our Cross-Encoder boosted Hit@10 from 31.71% to 34.15% (+2.44% absolute gain), proving that full token-to-token cross-attention rescues relevant documents that bi-encoders rank lower due to short-query lexical mismatch.
- Mean Reciprocal Rank (MRR): Evaluates how highly the first relevant chunk is ranked ($\frac{1}{|Q|} \sum \frac{1}{\text{rank}_1}$). Our pipeline achieves an MRR of ~0.299, meaning the first relevant technical excerpt surfaces at rank 3.3 on average.
- Latency vs. Accuracy Trade-Off: Hybrid retrieval retrieves 20 candidates in just ~38 ms, while neural reranking adds ~74 ms of CPU inference, bringing total retrieval latency to ~112 ms. This 74 ms budget pays for itself by filtering out false-positive keyword matches and feeding dense, high-precision context to Gemini.”
🎯 Strategic Interview Defense & Deep-Dive Soundbites
💡 "How do you explain the difference between Bi-Encoders and Cross-Encoders to an interviewer?"
“A bi-encoder (like BGE) processes the query and passage through the neural network independently, producing two isolated embedding vectors $u$ and $v$. Comparison is a fast dot product $\langle u, v \rangle$, which enables pre-computing embeddings and indexing them in sub-linear search structures like HNSW ($\mathcal{O}(\log N)$ lookup). However, it compresses the entire passage into a single vector, losing fine-grained word-level interactions.
A cross-encoder feeds the query and passage together into the network as a single input pair, allowing every query token to attend to every passage token across all transformer layers. It is drastically more accurate, but its inference cost is $\mathcal{O}(N)$ forward passes per query. That is why industry pipelines use bi-encoders for high-recall candidate generation ($\text{top-}K=20$) and cross-encoders for high-precision reranking ($\text{top-}5$).”
💡 "Why did you use Reciprocal Rank Fusion instead of normalizing and weighting scores?"
“Score normalization techniques like Min-Max or Z-Score assume stable, comparable underlying probability distributions. In hybrid search, cosine similarity is bounded in $[0, 1]$ and concentrated in a narrow band (e.g., $0.65 - 0.85$), whereas PostgreSQL's ts_rank_cd is an unbounded frequency score sensitive to document length and term density. A single outlier keyword match can distort the entire normalized distribution. RRF solves this by ignoring raw magnitudes and relying entirely on ordinal position ($\frac{1}{60 + \text{rank}}$). It is completely parameter-free with respect to score ranges and provably robust across diverse query types.”
💡 "Why did you choose BAAI/bge-small-en-v1.5 (384 dimensions, float32) over alternative embedding models?"
“I selected BAAI/bge-small-en-v1.5 based on a rigorous evaluation of retrieval performance, HNSW index RAM footprint, and query latency:
- Memory Footprint in PostgreSQL HNSW: In pgvector, HNSW stores vector data and bidirectional graph links ($m=16$) in RAM. In FP32, a 384-dim vector takes $1,536\text{ bytes}$, whereas 768-dim (BERT-base / BGE-base) takes $3,072\text{ bytes}$ and 1536-dim (OpenAI) takes $6,144\text{ bytes}$. By choosing 384 dimensions, the entire HNSW graph easily fits into PostgreSQL's shared buffer cache in RAM, avoiding expensive NVMe disk paging during approximate nearest-neighbour traversal.
- Distance Computation Speed: Vector cosine distance (
<=>) is an $\mathcal{O}(D)$ dot-product operation. Halving the dimensionality directly cuts the memory bus bandwidth and floating-point operations per graph hop by $50\%$.
- Instruction-Tuned Asymmetric Querying: BGE-v1.5 prepends a task instruction prefix (
'Represent this sentence for searching relevant passages: ') only to queries, allowing short search terms to map into the vector space of expository documentation while storing document chunks clean.
- Evaluation of Alternatives:
all-MiniLM-L6-v2 (384-dim): Popular lightweight baseline, but lacks modern contrastive hard-negative mining, yielding significantly lower retrieval recall on technical documentation.
bge-base-en-v1.5 (768-dim) / bge-large-en-v1.5 (1024-dim): Offers only +1–2% higher MTEB scores, but doubles or triples HNSW RAM consumption and vector math latency with diminishing returns, since our downstream cross-encoder already fixes fine-grained precision.
- OpenAI
text-embedding-3-small (1536-dim / 512-dim MRL): Proprietary cloud API that adds 100–300 ms network latency per query, per-token API costs, and breaks local offline execution in the terminal CLI.
intfloat/e5-small-v2 (384-dim): Requires explicit prefixes on both queries ('query: ') and stored documents ('passage: '), creating database storage bloat and coupling stored data to the embedding model. BGE leaves stored documents unmodified.
”
💡 "How do you evaluate retrieval quality in your RAG system, and what were your empirical findings?"
“I evaluated three retrieval configurations on a benchmark of 41 questions: vector-only search, hybrid search using Reciprocal Rank Fusion, and hybrid search with cross-encoder reranking.
Hybrid search did not show measurable improvement over vector-only retrieval in my current experiments. Adding cross-encoder reranking improved Recall@10 from 30.5% to 31.7% and Hit@10 from 31.7% to 34.1%. However, MRR decreased from 0.2988 to 0.2791, while retrieval latency increased from approximately 32 milliseconds to 112 milliseconds.
These results revealed a trade-off between retrieving more relevant chunks within the top ten results, ranking the first relevant result highly, and latency. My next step would be to investigate why hybrid retrieval provides no measurable gain and optimize the retrieval pipeline based on per-query results.”