Building a Reliable RAG

RAG (Retrieval-Augmented Generation - an artificial intelligence technique that improves large language model (LLM) responses by fetching relevant facts from an external knowledge base before generating an answer) sounds complicated until you stop thinking about it as “building an AI model.”  In practice, it is much closer to building a data pipeline for knowledge:

Prepare documents → structure data → create embeddings → retrieve → rerank → let an LLM explain → verify grounding → build chatbot.

After building this pipeline, I think the most important lesson is that the LLM is actually only one small part of the system. The quality of the answer starts much earlier  - with the data.

1. Start with the document, not the LLM

The first temptation is to download an LLM and start asking questions. For RAG, I would do the opposite.

Start by preparing your source.

A PDF is convenient for humans, but not necessarily for retrieval. First extract and clean the text while preserving its semantic structure. For a structured document, that might look like:

Document

 └── Chapter

      └── Section

           └── Subsection

Then decide what one retrievable unit - a chunk - should represent.

Instead of blindly splitting every 500 characters, I prefer using the natural structure of the document whenever possible. A section, rule, procedure or other meaningful unit can become one chunk; very long units can be split further while retaining their original metadata.

Each chunk can then look roughly like:

{

  "document": "document_name",

  "chapter": "chapter_name",

  "section": "section_number",

  "source_type": "primary_source",

  "text": "The actual source text..."

}

This is an important part of the recipe: don't throw away structure while preparing the data.

The metadata later lets us say not only “I found this text”, but also “this came from this document, chapter and section.”

I like JSONL for this stage because each line represents one independent knowledge unit and it remains easy to inspect, regenerate and debug.

2. Embeddings: turning meaning into something searchable

Now we have perhaps hundreds or thousands of chunks. We could search them with keywords. But users don't necessarily ask questions using the exact vocabulary appearing in the source. This is where an embedding model enters the pipeline.

An embedding model transforms text into a numerical vector representing aspects of its meaning:

"some piece of text"

        ↓

embedding model

        ↓

[0.021, -0.034, 0.118, ...]

For this experiment, I chose BGE-M3. The important reason wasn't that it is simply “a good embedding model.”  

I needed multilingual semantic retrieval, because the source and questions may not behave like English-only datasets.

So the same model embeds both:

document chunks → vectors

user question   → vector

We can then look for document vectors that are close to the question vector.

One useful distinction:

The embedding model doesn't answer the question. It helps us find text that may contain the answer.

3. What is ChromaDB?

Now we have vectors, but we need somewhere to store and search them. That's the role of ChromaDB.

Think of a normal database as something you might query using exact values: section = 12

A vector database lets us search by semantic similarity instead.

Our pipeline becomes:

JSONL chunks

     ↓

embedding model

     ↓

vectors

     ↓

ChromaDB

Chroma can keep the embedding together with the original text and its metadata. When a question arrives:

Question

   ↓

BGE-M3

   ↓

question vector

   ↓

ChromaDB

   ↓

most similar chunks

This is the retrieval part of RAG. And this stage is worth testing completely independently from the LLM.

Ask a question and inspect the retrieved chunks yourself.

If the correct source isn't being retrieved, adding a smarter LLM won't fix the fundamental problem.

4. Why add a reranker?

Semantic similarity gives us good candidates, but the nearest embedding isn't necessarily the chunk that best answers the question. That's why I added a second stage: reranking.

Suppose Chroma retrieves 15 candidates:

Question

   ↓

Vector search

   ↓

15 potentially relevant chunks

   ↓

Reranker

   ↓

5 strongest chunks

The embedding search is good at efficiently narrowing a large collection down to likely candidates.

The reranker then compares the question more directly against those candidates and changes their order according to relevance. So I think of the two jobs as:

Embedding retrieval: “Which pieces are probably related?”

Reranking: “Of those pieces, which ones are actually the best for this question?”

This two-stage approach is particularly useful for documents where many sections discuss closely related concepts.

5. Only now bring in the LLM

At this point, we finally need a generative model. But its job is deliberately limited.

It doesn't receive the entire PDF and it isn't supposed to search through our database. It gets something like:

USER QUESTION + TOP RETRIEVED AND RERANKED CHUNKS

Then we instruct it:

Answer using the supplied sources. If the sources don't contain enough information, say so.

This changes the architecture significantly.

Instead of:

Question → LLM → hope

we have:

Question

   ↓

Embedding

   ↓

ChromaDB

   ↓

Candidates

   ↓

Reranker

   ↓

Best source chunks

   ↓

LLM

   ↓

Natural-language answer

The LLM becomes an explanation layer over retrieved data, rather than our source of truth.

6. Grounding is another pipeline step

RAG reduces hallucination risk, but it doesn't magically eliminate it. An LLM can receive correct sources and still introduce a statement that isn't actually supported by them. So I added another concept to the recipe: grounding verification. Take:

Question + Retrieved sources + Generated answer

and check whether the claims in the answer are supported by those sources.

The objective isn't to declare an answer mathematically “true.” It's a quality-control step that can detect unsupported claims and allow the application to refuse or flag an answer when evidence is insufficient.

That gives us:

Retrieve → Rerank → Generate → Verify

This part feels very familiar from data engineering: don't just produce an output - validate it.

The reusable recipe

The nice thing is that none of this architecture is specific to one dataset.

You can repeat the same experiment with policies, technical documentation, research papers, internal procedures, regulations, manuals or another structured knowledge collection.

Don't start with the chatbot. Start with the data.

For me, that's the most interesting part of RAG: underneath the LLM, it is still very much a data problem - structure it well, retrieve it well, validate the output, and only then make it look intelligent.