4–5 Minute Detailed Architecture: RAG Technical Documentation Assistant

Target Duration: ~4 to 5 minutes (~800–950 spoken words)
Tone: Systems architect, deeply technical, precise on algorithmic and database trade-offs.
Structure: 5 Architectural Stages covering Ingestion/Chunking, Single-Store PostgreSQL/pgvector Indexing, Hybrid RRF Retrieval, Cross-Encoder Reranking, and Citation-Grounded Generation/Evaluation.


🎙️ Complete Architectural Presentation Script

“To explain the architecture and systems engineering behind this project, I break it down into five key stages: the ingestion and token-aware chunking pipeline, unified dual-index PostgreSQL database storage, hybrid retrieval with Reciprocal Rank Fusion, cross-encoder neural reranking, and citation-grounded LLM synthesis with benchmark evaluation.


Stage 1: Ingestion Pipeline & Recursive Token-Aware Chunking

The first challenge was preparing a heterogeneous technical corpus comprising official PostgreSQL 16 HTML documentation and Linux system manual pages.

In standard RAG tutorials, documents are sliced into naive character-based or fixed-line windows. On technical manuals, this causes disastrous splits: code snippets get sliced mid-statement, command flags become separated from their explanations, and table headers are orphaned. I built a multi-stage parser:

(Hook: Sets up deep-dive into chunk boundary strategies and embedding batching)


Stage 2: Unified Dual-Index Database Engine (PostgreSQL + pgvector)

“A pivotal design decision was avoiding external dedicated vector databases like Pinecone, Qdrant, or Milvus in favor of a single-store architecture in PostgreSQL 16 using pgvector.

In production systems, running a separate vector database alongside a relational database introduces severe distributed systems overhead: dual-write transaction failures, replication lag between metadata and vectors, and complex network topologies. PostgreSQL satisfies all our requirements in one ACID-compliant engine:

(Hook: Sets up questions on HNSW graph memory footprint vs. IVFFlat list probing)


Stage 3: Hybrid Retrieval & Reciprocal Rank Fusion (RRF)

“During retrieval, relying solely on dense vector embeddings creates significant blind spots for technical documentation. Vector models capture semantic concepts well—like asking 'how does concurrency work?'—but they frequently fail on exact keywords, command-line flags (such as grep -E versus grep -F), or specific configuration variables like max_connections.

To overcome this, I built a hybrid retrieval engine that queries both indexes in parallel:

(Hook: Sets up discussion on score normalization vs. rank aggregation)


Stage 4: Cross-Encoder Neural Reranking

“Even after RRF fusion, bi-encoder retrieval has an intrinsic limitation: bi-encoders map query and document independently into fixed-dimensional vectors, meaning all term interaction is compressed into a simple dot product.

To maximize precision before passing context to the LLM, I integrated a Cross-Encoder reranking model (cross-encoder/ms-marco-MiniLM-L-6-v2):

(Hook: Sets up deep-dive into computational complexity and two-stage search pipelines)


Stage 5: Citation-Grounded LLM Generation, Serving & Empirical IR Evaluation

“The final stage is grounded answer synthesis, serving, and quantitative evaluation:


🎯 Strategic Interview Defense & Deep-Dive Soundbites

💡 "How do you explain the difference between Bi-Encoders and Cross-Encoders to an interviewer?"
“A bi-encoder (like BGE) processes the query and passage through the neural network independently, producing two isolated embedding vectors $u$ and $v$. Comparison is a fast dot product $\langle u, v \rangle$, which enables pre-computing embeddings and indexing them in sub-linear search structures like HNSW ($\mathcal{O}(\log N)$ lookup). However, it compresses the entire passage into a single vector, losing fine-grained word-level interactions.

A cross-encoder feeds the query and passage together into the network as a single input pair, allowing every query token to attend to every passage token across all transformer layers. It is drastically more accurate, but its inference cost is $\mathcal{O}(N)$ forward passes per query. That is why industry pipelines use bi-encoders for high-recall candidate generation ($\text{top-}K=20$) and cross-encoders for high-precision reranking ($\text{top-}5$).”
💡 "Why did you use Reciprocal Rank Fusion instead of normalizing and weighting scores?"
“Score normalization techniques like Min-Max or Z-Score assume stable, comparable underlying probability distributions. In hybrid search, cosine similarity is bounded in $[0, 1]$ and concentrated in a narrow band (e.g., $0.65 - 0.85$), whereas PostgreSQL's ts_rank_cd is an unbounded frequency score sensitive to document length and term density. A single outlier keyword match can distort the entire normalized distribution. RRF solves this by ignoring raw magnitudes and relying entirely on ordinal position ($\frac{1}{60 + \text{rank}}$). It is completely parameter-free with respect to score ranges and provably robust across diverse query types.”
💡 "Why did you choose BAAI/bge-small-en-v1.5 (384 dimensions, float32) over alternative embedding models?"
“I selected BAAI/bge-small-en-v1.5 based on a rigorous evaluation of retrieval performance, HNSW index RAM footprint, and query latency: ”
💡 "How do you evaluate retrieval quality in your RAG system, and what were your empirical findings?"
“I evaluated three retrieval configurations on a benchmark of 41 questions: vector-only search, hybrid search using Reciprocal Rank Fusion, and hybrid search with cross-encoder reranking.

Hybrid search did not show measurable improvement over vector-only retrieval in my current experiments. Adding cross-encoder reranking improved Recall@10 from 30.5% to 31.7% and Hit@10 from 31.7% to 34.1%. However, MRR decreased from 0.2988 to 0.2791, while retrieval latency increased from approximately 32 milliseconds to 112 milliseconds.

These results revealed a trade-off between retrieving more relevant chunks within the top ten results, ranking the first relevant result highly, and latency. My next step would be to investigate why hybrid retrieval provides no measurable gain and optimize the retrieval pipeline based on per-query results.”

← 1–2 Minute Speech Next: Complete End-to-End Flow →