Building RAG Systems That Actually Find the Right Information

Everyone's first instinct with retrieval-augmented generation is to focus on the model. Which LLM, which prompt, which temperature. I spent a long time there too, before I figured out the thing nobody tells you up front: almost none of your quality problems live in the model. They live in retrieval, the step before the model ever sees a word. If you hand a good model the wrong passages, it will write a fluent, confident, wrong answer every single time. Here's what I actually learned building a system that has to get this right, because it's answering questions from books that don't have reliable citations anywhere else on the internet.

Vectors: what the machine is actually comparing

The first real unlock was understanding what a vector even is, because every other concept in RAG is built on top of it.

Once that clicked, the rest of RAG stopped being magic and started being geometry. You're not asking a database "find this text." You're asking it "find me the coordinates closest to this other coordinate."

Chunking: the size that's fast and accurate is never the size that's obviously right

Before any of that geometry works, you have to decide how to cut your source text into pieces, chunks, small enough to search quickly and precisely, large enough to still mean something on their own. This turned out to be one of the most consequential decisions in the entire system, and there's no universal right answer.

A small chunk, a sentence or two, is sharp and fast to search, but it's often missing the surrounding idea that gives it meaning. A large chunk, a full page, carries plenty of context, but the vector that represents it gets vague, because it's trying to describe everything in the chunk at once instead of any one thing precisely.

The part that actually surprised me: a chunk can be simultaneously wrong and right, depending on the question. A narrow, three-sentence chunk might be completely out of context against the larger chapter it was pulled from, and still be exactly the right, pinpointed answer for a narrow, specific question. Context isn't a property of the chunk by itself. It's a relationship between the chunk and the question being asked of it. That's part of why there's no single "correct" chunk size, only a size that's right for the kinds of questions you expect to get.

Retrieval: finding the handful of chunks worth reading

Retrieval is the step that takes a question, turns it into a vector, and asks the database for the chunks whose vectors sit closest to it. Out of potentially tens of thousands of chunks, retrieval is responsible for narrowing that down to the handful that might actually contain the answer, before the model ever gets involved. If retrieval hands the model the wrong ten chunks, nothing downstream can fix that. The model can only reason about what it was shown.

Embeddings: the step that turns text into the vector in the first place

Vectors don't appear out of nowhere. Something has to read a chunk of text and produce its coordinate, and that something is called an embedding model. I'd used the word "embedding" for a while before I actually understood it was the process, not the end product.

Why it matters so much in practice: the embedding model you choose determines the entire shape of your search space. A model tuned for general web text understands "the great liner" differently than one tuned for literary or technical text. Get the wrong embedding model for your domain, and you get confidently wrong neighbors, chunks that are mathematically close but conceptually unrelated, and no amount of clever code downstream fixes that.

Hybrid search: vectors aren't the whole story

Pure vector search is great at meaning and genuinely bad at exact wording. Ask about a phrase that's mentioned exactly once in a 300,000-word book, and a vector search can miss it completely, because that one vivid sentence barely moves the average meaning of the whole chunk it lives in. The fix was running a second, completely different kind of search alongside the vector one: old-fashioned keyword search, the kind that rewards exact words and rare terms. Then fusing the two rankings together. The vector arm finds paraphrase and synonymy. The keyword arm finds the exact, rare, single-mention phrase. Running both and merging the results catches what either one misses alone.

Reranking: the first results aren't always the best ones

The next surprise was that whatever order retrieval hands back isn't necessarily the right order for answering the actual question. The search step is fast and approximate on purpose, it's scanning a huge number of candidates and can't afford to think too hard about any single one. So I added a second, slower pass that actually reads the question next to each candidate passage and re-sorts them by genuine relevance.

Prompt engineering: the part everyone starts with, and the part that moves the least

I did eventually spend real time on prompt engineering: how to phrase the model's instructions, how to structure the output I wanted back, and how to adjust the wording when results came back in a shape I didn't expect, too chatty, too hedgy, citing the wrong section. It matters, and it's worth tuning. But it was never the lever that fixed a wrong answer. Every meaningful quality jump in this system came from retrieval, chunking, hybrid search, reranking, never from a cleverer sentence in the system prompt. Prompting changes tone. It doesn't rescue bad passages.

Context: what you feed the model besides the question

Once retrieval and reranking hand back the right passages, the next job is assembling everything the model actually needs to see before it answers.

The retrieved passages are the obvious part. The less obvious part is deciding how much of the prior conversation belongs in that context too. Sometimes a follow-up question genuinely depends on the last answer, "what about the second one," and needs that history to make sense at all. Other times a user jumps to a completely unrelated question mid-conversation, and dragging the old topic along just confuses retrieval, because now you're searching for a blend of two unrelated questions instead of one clear one. Deciding when a question is a continuation versus a fresh start, and only carrying history forward when it's actually a continuation, turned out to be its own small classification problem sitting in front of the real one.

Re-running prompts when the first answer isn't good enough

Not every first attempt is usable. Sometimes the retrieved context is too thin, too few relevant chunks came back, and sometimes it's too broad, the question matched too many tangentially related passages and the model's answer wanders. I added an internal quality check that scores a draft answer before it ever reaches the user, based on things like how many citations came back verified, how tightly the retrieved passages clustered around the question, and whether the draft actually used what it was given. A low score triggers one corrective retry, often with a reformulated search, before deciding whether to answer or refuse outright. That internal ranking step, grading your own output before showing it to anyone, caught more bad answers than any single upstream improvement.

LangChain, tools, and guardrails

At some point the system stopped being "retrieve, then ask the model once" and became a small agent that can take a few actions: search again with different wording, look up a term in a glossary, or call a dedicated safety tool. One of the tools I built is specifically a "Risk to Life" check, a classifier that runs before anything else when a question suggests someone might be in crisis, routing to an appropriate, careful response instead of letting the main answering pipeline treat it like an ordinary question. That's a guardrail, not a feature: a rule that exists to make sure the system never treats a serious situation like a trivia lookup.

Steering: keeping the conversation on one subject

Left alone, a capable model is genuinely eager to help, which sounds good until it starts pulling in information from well outside the book it's supposed to be answering from, because it thinks a tangent might be "useful." I had to explicitly steer it back: scoping its instructions so tightly to the source material that going off on a helpful-sounding tangent isn't a style choice it's allowed to make. LLM steering, in practice, is less about telling the model what to say and more about narrowing what it's even allowed to consider saying.

Reducing hallucinations by making the model check itself

The single biggest hallucination reduction didn't come from a better prompt, it came from forcing the model to verify its own output against the stored source text before anything reaches the user. After the model drafts an answer with quoted passages and citations, a separate, deterministic check confirms every quote actually exists, word for word, in the passage it claims to come from. If a quote doesn't check out, that's treated as invention, not a stylistic quirk, and the system either retries or refuses rather than publish it. Asking a model to grade its own honesty is asking the wrong judge. Checking its claims against the actual source text is not.

The result

All of this came out of a very specific, very stubborn problem: studying books that don't have reliable citations or discussion anywhere else online, where the usual trick of "let the model recall what it already knows" produces fluent, confident, and sometimes simply false answers. What I ended up building is a system that reads the book itself every time, chunked and indexed carefully enough to find a single rare phrase, searched two different ways and reranked by an actual reader, answered with citations that are mechanically checked against the source before anyone sees them, and steered tightly enough to stay on the one subject it's actually qualified to discuss. Someone using it can trust an answer the way they'd trust a footnote, because it's been checked the same way a footnote would be, while it still feels like an ordinary conversation.

← Back to writing Get in touch