Retrieval-Augmented Generation, Explained for Busy Engineers

Large language models are trained on a snapshot of the public internet. They have never seen your contracts, your product docs or last week's support tickets, and fine-tuning them on that material is slow, expensive and goes stale the moment the data changes.

Retrieval-augmented generation (RAG) takes a different route: instead of teaching the model your data, you look up the relevant pieces at question time and hand them to the model as context.

The pipeline in four steps

1. Chunk. Split your documents into passages of a few hundred tokens. Keep headings with their paragraphs so each chunk still makes sense on its own.
2. Embed. Run every chunk through an embedding model, which turns text into a vector: a list of numbers where similar meanings sit close together.
3. Retrieve. When a question comes in, embed it the same way and fetch the chunks whose vectors are nearest to it.
4. Generate. Put those chunks into the prompt with an instruction like "Answer using only the context below, and say so if the answer isn't there."

const queryVector = await embed(question);
const chunks = await vectorStore.search(queryVector, { topK: 5 });

const answer = await llm.generate({
  system: "Answer only from the provided context. Cite the source of each claim.",
  prompt: `Context:\n${chunks.map((c) => c.text).join("\n---\n")}\n\nQuestion: ${question}`,
});

Where RAG systems actually fail

In our experience the model is rarely the weak link. Retrieval is.

• Bad chunking. A chunk that starts mid-table or loses its heading retrieves badly and reads worse. Chunk on document structure, not a fixed character count.
• Vocabulary mismatch. Users ask about "leave days"; the policy says "annual entitlement". Pure vector search helps, but combining it with keyword search (hybrid search) catches exact terms like product codes and names.
• Too much context. Stuffing twenty chunks into the prompt buries the answer. Retrieve generously, then re-rank and keep the best few.
• Stale indexes. If documents change and embeddings don't, answers drift. Re-embed on update, not on a monthly cron.

Measure it before you tune it

Build a small evaluation set early: 30–50 real questions with known answers and the documents that contain them. Track two things separately:

Metric	Question it answers
Retrieval recall	Did the right chunk make it into the top k?
Answer faithfulness	Did the answer stick to the retrieved context?

If recall is low, no prompt will save you; fix chunking and search first. If recall is high but answers are wrong, work on the prompt and the model.

When not to use RAG

If your data fits comfortably in the model's context window and rarely changes, just include it. If you need the model to adopt a style or format rather than know facts, fine-tuning or better prompting is the right tool. RAG is for large, changing knowledge that has to be cited.

Done well, RAG is the most practical way to put an LLM to work on private data, and the engineering is mostly good old search.