RAG Development: Building Retrieval That Holds Up in Production

Retrieval-augmented generation (RAG) is the most common way to make a large language model answer questions about your own documents, tickets, contracts or product data. The idea is simple: find the relevant passages, put them in the prompt, and ask the model to answer from them. A working prototype takes an afternoon.

Getting the same system to answer correctly for thousands of users, over messy data, with permissions and a budget, is a different job. This guide covers the decisions that matter in RAG development once you move past the demo — the same checklist we work through on client projects.

Start with the questions, not the vector database

Before choosing a model or a store, collect 50–100 real questions your users ask, along with where the answer lives. This set becomes your evaluation baseline, and it tells you a lot about the shape of the problem. Are questions mostly factual lookups, comparisons across documents, or summaries? Do they reference product codes and IDs that semantic search handles badly? Are some documents confidential to certain teams? Each answer changes the architecture.

Chunking: where most quality problems begin

Chunking decides what the retriever can find. Fixed-size chunks of a few hundred tokens are a fine starting point, but they break tables, split procedures in half and strip away the heading that gave a paragraph its meaning. In practice we:

• Chunk along document structure — headings, sections, list items, table rows — and fall back to size limits only when a section is too long.
• Carry context into every chunk: document title, section path, date and source system, so a chunk still makes sense on its own.
• Use modest overlap between neighbouring chunks so a sentence that straddles a boundary is not lost.
• Treat tables and PDFs as a separate parsing problem. Poor extraction upstream cannot be fixed by a better model downstream.

Embeddings and hybrid search

Dense embeddings are good at meaning and bad at exact strings. A user searching for an error code, a SKU or a clause number needs keyword matching. That is why most production RAG systems we build use hybrid search: a vector index for semantic similarity plus a lexical index (BM25 or similar) for exact terms, with results merged by a method such as reciprocal rank fusion.

Pick an embedding model that handles the languages you actually have — for clients across Germany, Switzerland and Qatar that often means German, French and Arabic alongside English — and version it. Changing the embedding model means re-embedding the corpus, so record which model produced each vector. On Dorfleben, a community platform for towns across Germany, AI-powered search and recommendations ran on Weaviate with models served through SageMaker and Ollama, which is a good example of matching the retrieval stack to the content and language of the users.

Reranking and the context window

Retrieval should cast a wide net — say the top 30–50 candidates — and a reranker should choose the handful that go into the prompt. Cross-encoder rerankers read the question and the passage together, so they are much better at judging relevance than embedding similarity alone. Sending fewer, better passages to the model reduces cost, latency and the chance of the model latching onto an irrelevant one.

async function retrieve(query: string, user: User) {
  const filter = { tenantId: user.tenantId, groups: user.groups };

  const [semantic, keyword] = await Promise.all([
    vectorIndex.search(await embed(query), { topK: 40, filter }),
    keywordIndex.search(query, { topK: 40, filter }),
  ]);

  const candidates = reciprocalRankFusion([semantic, keyword]);
  const ranked = await rerank(query, candidates);

  return ranked.slice(0, 6); // only the best passages reach the prompt
}

Note that the permission filter is applied inside both searches, not after. That detail matters more than any model choice.

Access control, citations and refusals

A RAG system must never retrieve a document the user could not open themselves. Store access metadata with every chunk, sync it from the source system, and filter at query time. Post-filtering the answer is too late: the model has already read the restricted text.

Every answer should cite its sources with links back to the original document and section. Citations let users verify, let your team debug, and give the model a reason to stay grounded. Equally important is a clear instruction — and a tested behaviour — for saying “I could not find this” when retrieval comes back empty or weak. A confident wrong answer does far more damage than an honest gap.

Evaluation: measure retrieval and generation separately

When a RAG answer is wrong, the cause is usually retrieval: the right passage never reached the model. So evaluate the two stages separately:

1. Retrieval quality — for each question in your test set, did the expected source appear in the top results? Track recall and rank.
2. Answer faithfulness — is every claim in the answer supported by the retrieved passages?
3. Answer usefulness — does it actually address the question? Use human review on a sample, supported by an LLM judge for scale.
4. Regression runs — re-run the suite on every change to chunking, prompts, models or data pipelines, and block releases that make it worse.

Log real queries in production (with appropriate privacy controls), sample the failures each week and add them to the test set. The evaluation set should grow with the product.

Keeping the index fresh

Stale data is a quiet failure mode. If a policy changes and the old version stays in the index, the system will cite it with confidence. Connect ingestion to the source systems through change events or scheduled syncs, delete chunks when documents are removed or permissions change, and show the last-updated date next to each citation so users can judge for themselves. Monitor ingestion like any other pipeline: failed documents, parsing errors and sync lag should raise alerts, not wait for a user complaint.

Keeping cost and latency under control

RAG costs come from embedding, storage, reranking and generation. The levers are practical: cache embeddings and frequent answers, keep prompts short by sending fewer, better chunks, route simple questions to a smaller model, and stream responses so users see progress. Incremental indexing — only re-embedding documents that changed — keeps ingestion cheap as the corpus grows. For sensitive data, self-hosted open models (we have used Ollama in production work, for example on Sumus) can keep inference inside your own infrastructure.

Building RAG with Amygdal

We are a software development company that builds AI into real products for clients in the UK, Germany, Switzerland and Qatar. Our RAG development work covers ingestion pipelines, hybrid retrieval, evaluation suites, permissions and the operations around them. If you are weighing whether RAG or a broader custom AI development project fits your problem, get in touch and we will look at your data and questions with you.