Building a RAG Pipeline From Scratch: The Three Fixes That Actually Matter

I built a production-grade RAG pipeline from scratch. No LangChain wrappers — raw components, every layer visible.

The goal wasn't to ship fast. It was to understand why each piece exists, because most RAG tutorials hand you a VectorStoreRetriever and call it done. That retriever is exactly the thing that breaks in production.

Here's what naive RAG gets wrong, and the three fixes that actually matter.

The problem with standard RAG

Chunk → embed → store → cosine similarity at query time.

Simple. Also broken in subtle ways.

Each of the three fixes below targets a different failure: the embedding losing context, dense vectors missing exact terms, and bi-encoders never comparing the query to the chunk directly.

Fix 1: Contextual Retrieval

The failure: bi-encoders produce context-free embeddings.

A chunk reading "It delegates to the parent using super()" encodes as a generic statement about delegation. Zero signal that it's about Java inheritance, in Chapter 5, of a specific textbook. At query time, "how does constructor chaining work in Java" won't find it — the vector simply doesn't carry that information.

The fix (from Anthropic's Contextual Retrieval work, Oct 2024): before embedding, pass (full_document, chunk) to a lightweight LLM and have it write 1–2 sentences situating the chunk. Prepend that to the chunk text, then embed.

The prompt that does it:

def _build_context_prompt(document_text: str, chunk_text: str) -> str:
    return f"""<document>
{document_text}
</document>

Here is a chunk from this document:
<chunk>
{chunk_text}
</chunk>

Write 1-2 sentences that situate this chunk within the overall document. \
Mention the topic, section, or concept this chunk belongs to. \
Do not summarize the chunk itself — only provide context that would help \
someone searching for this information. Answer with only the context sentences."""

Two details that matter more than they look:

temperature: 0.0 — deterministic. The same document and chunk always produce the same context, which is what makes the cache below sound.

num_predict: 50 — 1–2 sentences is 30–40 tokens. Capping it stops the small model from rambling into the embedding.

I used qwen2.5:0.5b for this. It only has to write two sentences, so a 0.5B model is plenty — and it's fast enough to run over an entire corpus.

The vector now encodes origin + concept, not just concept.

The part nobody mentions: make it incremental

Contextualising is one LLM call per chunk. Do that naively and every re-index costs you the whole corpus again.

So: hash each chunk (MD5 of raw text), and cache hash → generated context. On a re-run, unchanged chunks are instant cache hits.

def _chunk_hash(chunk_text: str) -> str:
    """MD5 of the raw chunk text — used as cache key."""
    return hashlib.md5(chunk_text.encode()).hexdigest()

The practical result:

• Add a document → only its new chunks pay the LLM cost. The existing 567 are free.
• Edit a document → only the chunks whose text actually changed get a new hash.
• No changes → the entire contextualiser stage is instant.

The calls themselves are blocking HTTP — pure I/O wait — so they run through a ThreadPoolExecutor with WORKERS = 4, matched to OLLAMA_NUM_PARALLEL=4. Without that flag Ollama serialises the requests and the pool buys you nothing.

Fix 2: Hybrid Search + Reciprocal Rank Fusion

The failure: pure dense retrieval misses lexical matches.

Query: "what does AbstractBeanFactory do". Semantic search returns related factory patterns — conceptually close, not what was asked. BM25 finds the exact string immediately.

Neither is strictly better. So run both, and merge the ranked lists with Reciprocal Rank Fusion:

score(d) = 1/(rank_semantic + k) + 1/(rank_bm25 + k)

def _rrf_merge(semantic_hits, bm25_hits, top_k):
    scores, payload = {}, {}

    for rank, hit in enumerate(semantic_hits):
        cid = hit["id"]
        scores[cid]  = scores.get(cid, 0.0) + 1.0 / (rank + 1 + RRF_K)
        payload[cid] = hit

    for rank, hit in enumerate(bm25_hits):
        cid = hit["id"]
        scores[cid]  = scores.get(cid, 0.0) + 1.0 / (rank + 1 + RRF_K)
        payload[cid] = hit

    ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)[:top_k]
    return [{**payload[cid], "rrf_score": round(s, 6)} for cid, s in ranked]

Why RRF rather than blending the scores directly:

It operates on ranks, not raw scores. Cosine distances and BM25 scores live on completely different scales with different distributions. Normalising them is comparing apples to oranges, and any weighting you pick is a guess. Ranks sidestep the problem entirely.

k = 60 damps the top. It's the constant from the original 2009 paper, and it stops a rank-1 hit from dominating. The effect: a document ranked #3 in both lists beats one ranked #1 in only one. Agreement between two independent retrievers is a stronger signal than a single confident vote.

Fix 3: Cross-encoder Re-ranking

The failure: bi-encoders encode query and document independently, so the relevance signal between them is never computed.

A cross-encoder feeds [query, SEP, chunk] through a transformer jointly — every query token attends to every chunk token. It reasons about relevance instead of measuring the distance between two vectors that never met.

That's why it catches matches like "doesn't call super()" ↔ "automatically inserts a call to the superclass constructor", where there's almost no lexical overlap and the embeddings sit far apart.

The catch is cost. It must run on every (query, chunk) pair, at query time:

20ms × 567 chunks ≈ 11 seconds per query   ← unusable
20ms ×  20 chunks ≈ 400ms per query        ← fine

So, two-stage retrieval. Cheap-and-approximate narrows the field; expensive-and-precise ranks what survives:

BM25 + semantic → RRF → top-20 → cross-encoder → top-5

pairs  = [(question, c["text"]) for c in candidates]
scores = reranker.predict(pairs)

for candidate, score in zip(candidates, scores):
    candidate["reranker_score"] = round(float(score), 4)

reranked = sorted(candidates, key=lambda x: x["reranker_score"], reverse=True)
top      = reranked[:top_k]

The re-ranker is the final arbiter — its scores override RRF order completely. RRF's only job is deciding which 20 chunks are worth the 400ms.

Knowing when to say nothing

This is the piece I'd argue matters most, and it's the one naive RAG never has.

The MS-MARCO cross-encoder emits a calibrated relevance score, which means you can threshold it:

Score	Meaning
> 0	genuinely relevant
-3 to 0	weakly related
< -3	not relevant

# If the best chunk doesn't clear the threshold, nothing is relevant.
if top and top[0]["reranker_score"] < RERANKER_THRESHOLD:
    return []

If the best available chunk scores below -3.0, retrieval returns nothing at all. In private mode the LLM never even sees the question — the system answers "not in your documents" directly.

Cosine similarity can't do this. It always returns your top-5, because something is always nearest. Ranking is relative; the cross-encoder score is absolute. That distinction is the difference between a system that says "I don't know" and one that confidently invents an answer from the five least-irrelevant chunks it could find.

Three retrieval modes

Retrieval isn't always what you want, so the mode is explicit:

Mode	Retrieval	LLM behaviour	Use when
public	none	unrestricted — training knowledge only	general topics the model knows better than your docs
private	full pipeline	strict — only indexed documents, refuses otherwise	internal docs, private data, anything past the training cutoff
hybrid	full pipeline	free — context and training knowledge	default

The threshold gate only applies to private and hybrid. In private it produces a refusal; in hybrid it quietly falls back to the model's own knowledge.

The pipelines end to end

Indexing, which runs once (and incrementally after that):

refs/*.pdf / *.txt
       │
       ▼
   loader          Extract text from PDFs and text files
       │
       ▼
   chunker         Fixed-size sliding window (200 words, 40 overlap)
       │
       ▼
 contextualizer    LLM writes 1-2 context sentences per chunk
 (qwen2.5:0.5b)    Prepended to chunk text before embedding
 + MD5 cache       Only new/changed chunks pay the LLM cost
       │
       ▼
   embedder        all-MiniLM-L6-v2 → 384-dim vectors
       │
       ▼
  vectorstore      ChromaDB (persistent, SQLite-backed)

Query, which runs every turn:

                     User question
                          │
              ┌───────────┴───────────┐
              ▼                       ▼
      Semantic search           BM25 search
      (ChromaDB, dense)         (keyword, sparse)
        top-20                    top-20
              │                       │
              └───────────┬───────────┘
                          ▼
                    RRF merge  (k=60)
                       top-20
                          │
                          ▼
              Cross-encoder re-ranker
              (ms-marco-MiniLM-L-6-v2)
              threshold gate: < -3.0 → reject
                       top-5
                          │
                          ▼
                 Prompt construction
              (public / private / hybrid)
                          │
                          ▼
              Ollama: dolphin-llama3  (streamed)

The stack

Everything runs locally. No cloud APIs, no per-token cost.

Component	Choice	Why
LLM	dolphin-llama3 via Ollama	local, no API cost
Contextualiser	qwen2.5:0.5b	only needs to write two sentences
Embeddings	all-MiniLM-L6-v2	384-dim, fast, runs on CPU
Re-ranker	cross-encoder/ms-marco-MiniLM-L-6-v2	fine-tuned on 1M real search queries
Vector DB	ChromaDB	local, SQLite-backed, zero setup
Keyword search	rank-bm25	BM25Okapi, in-memory
Cache + memory	Redis	context cache, conversation history, index status
PDF parsing	pypdf	pure Python, no system deps

The knobs

Every number in this post is a constant you can move:

Constant	Default	Effect
CANDIDATES_K	20	candidates per retriever before RRF
RRF_K	60	RRF damping — higher flattens rank-1 advantage
RERANKER_THRESHOLD	-3.0	minimum score to count as relevant
TOP_K	5	final chunks injected into the prompt
WORKERS	4	parallel Ollama calls while indexing
DOC_TRUNCATE_WORDS	1000	document words passed to the contextualiser
NUM_CTX	2048	contextualiser context window

If you tune one thing, make it RERANKER_THRESHOLD — it's the dial between "refuses too often" and "answers when it shouldn't", and the right value depends entirely on your corpus.

What's next

• Dockerise the indexer and chatbot as separate services
• Groq API support for much faster contextualisation
• Sentence-window retrieval
• Query expansion / HyDE

Takeaway

RAG is an information retrieval problem wearing an LLM jacket.

Everyone focuses on the generation half — prompt templates, model choice, temperature. But if retrieval hands the model the wrong five chunks, no amount of prompt engineering saves the answer. The retrieval engineering is where the real work is.

The code is on GitHub, including the FastAPI server with SSE streaming and the web UI:

👉 GitHub — GenericChatBot