RAG Isn't Magic, It's Search Engineering Most Teams Get Wrong

A client showed us a demo of their "bot that knows everything about our company." We asked it one concrete thing: the return window in their terms of service. The bot confidently said 30 days. The document said 14. Nobody in the room noticed until we opened the PDF and checked. That's RAG for most teams in a nutshell: it sounds smart and gets it wrong exactly where it can't afford to.

The pitch goes like this. Drop your documents into a vector database, connect an LLM, and you get a smart assistant that answers from your own data. Technically true. In practice, that naive approach, dump the PDFs and run a cosine similarity search, is a prototype at best in 2026 and a liability at worst. The problem is almost never the model. The problem is a search step that hands the model the wrong pieces of text, and then the model politely hallucinates on top of garbage.

Retrieval is the real problem, not generation

RAG has two parts. Retrieval, the part that finds relevant pieces from your documents, and generation, the part where the LLM writes the answer. Everyone talks about the model because it's the flashy part. But if retrieval pulls the wrong paragraphs, the best model on earth can't save the answer. It gets bad material, it writes a bad answer, and it does so with total confidence.

Here's the most common place it breaks. Vector search works on semantic similarity. It's great when a user asks "how do I cancel my subscription" and the document talks about "terminating the contract." But that same search is weak when the user wants an exact term. A product SKU, a function name, a statute number, a version string. The embedding sees that "v2.3.1" and "v2.4.0" are very similar, because they are, semantically. For a user asking for exactly 2.3.1, that's the wrong answer. Pure vector search routinely misses what plain keyword search would nail on the first try.

Chunking is the decision everyone skips

Before anything enters the database, the document gets cut into pieces. That step, chunking, is the one teams usually leave on default and forget. Yet it determines quality as much as the choice of embedding model. Cut the pieces too small and you lose context. The sentence "this does not apply to business customers" is meaningless if it's severed from the clause it refers to. Cut them too large and you feed noise into search, half a page of irrelevant text around the one line that matters, and the model gets lost.

On one project where we inherited a finished RAG setup, everything was "by the tutorial": fixed chunks of a few thousand characters, no overlap. The result was that tables and lists were being split mid-row. A price lived in one chunk, the condition attached to that price in the next, and the model never saw both together. We moved to structure-aware splitting, chunks around 500 tokens with a small overlap, and special handling for tables. That fixes far more than swapping in a more expensive model.

How to fix it without any magic

First, hybrid search. Combine classic keyword search (BM25, the same logic every serious search engine uses) with vector search. The keyword side catches exact terms, the vector side catches meaning. You merge the results. This is the biggest quality jump for the least effort and it should be the default, not an advanced technique.

Second, reranking. Instead of feeding the model the top five search results directly, pull twenty, then let a separate, more precise model reorder them by real relevance and hand the LLM only the best. This step costs a little latency and throws out a surprising amount of almost-right-but-wrong chunks that would otherwise fool the model.

Third, and this is what separates a toy from a product: measurement. If you don't test retrieval against real questions with known correct answers, you're just hoping. Build a set of realistic user questions, define which chunk of the document holds the correct answer, and measure how often your search retrieves that chunk at all. Without this, every change is a shot in the dark, and "seems better now" is not a metric.

RAG isn't magic and it never was. It's search engineering with an LLM on the end. Companies that get this build assistants users trust. The ones who think dropping in PDFs is enough build a system that confidently says 30 days when the answer is 14. The difference isn't the model. The difference is whether you understand you were building a search engine all along.

Related:

• 84% of Developers Use AI Tools. So Why Isn't the Code Better?
• AI in Web Development 2025: Where It Actually Helps and Where It's Just Hype

Building an internal assistant or a chat-with-your-documents bot and not sure the retrieval is good enough? Contact us and we'll look at where your search is breaking.