RAG Explained: Build a Private Document Q&A App You Can Debug and Trust

Start with the question, not the database

RAG is often introduced as a three-step recipe: split documents, create embeddings, retrieve chunks. That recipe is incomplete. Before choosing a database, decide what question the system must answer and what evidence would make the answer acceptable.

Imagine a small internal handbook assistant. A user asks, “Can I expense a train ticket purchased two weeks after the trip?” A useful system must find the relevant policy version, preserve the exception and date rules, cite the source, and avoid answering if the handbook does not cover the case.

That is a product and data problem before it is a model problem. Write ten questions first. For each one, record the expected source, the important claim, and whether the correct behavior is an answer, a clarification, or an abstention.

The mental model: evidence moves through five gates

Think of a RAG request as a sequence of gates. Each gate can fail independently.

Debug the gates in this order. Do not jump straight to a larger model when the source text was never extracted correctly.

Define the private boundary before uploading anything

“Private” is not a model feature. It is an application boundary.

For every document, store an owner, workspace or tenant, version, source location, and access group. Apply authorization before retrieval, not after the model has already seen the context. The model should receive only the chunks the current user is allowed to access.

Use synthetic or public documents while learning. Do not use employer files, customer records, credentials, or personal documents merely to make a demo look realistic.

Build the ingestion pipeline as its own product

Retrieval cannot recover information that ingestion destroyed. Treat ingestion as a visible pipeline:

file -> type check -> text extraction -> structure repair -> metadata -> chunks -> index

Keep the original source separate from the extracted representation. Save the parser version and content hash so you can tell whether a retrieval change came from a new index or a changed document.

Test difficult documents early

• A two-column PDF
• A document with headings and footnotes
• A table whose row labels carry meaning
• A scanned page that requires OCR
• A revised policy with an older version nearby

Inspect extracted text manually. If a human cannot recognize the important facts in the extracted output, embedding it only makes the broken representation easier to search.

Chunking is an experiment, not a constant

Chunking decides what a retrieval result can contain. A chunk that is too small loses context; one that is too large brings irrelevant material into the answer.

Run the same question set against at least two strategies. Record which expected source was retrieved, how much irrelevant context appeared, and whether the answer could be supported.

Design retrieval for relevance and authorization

A useful retrieval query usually combines semantic similarity with ordinary constraints.

candidates = search(
  queryEmbedding,
  where: { workspaceId, accessGroup, documentStatus: 'current' },
  limit: 12
)
context = rerank(candidates).slice(0, 5)

The exact APIs differ, but the order matters: authorization and document state are filters, not suggestions to the model. Similarity search should find candidates; reranking or a second pass can improve precision.

Try keyword plus vector search when users ask for exact policy names, error codes, product identifiers, or dates. Try query rewriting when the user’s conversational question does not contain the terms used in the document. Measure each change against the fixed question set rather than trusting a single impressive answer.

Give the answer an evidence contract

The generation step should receive a small, labeled context and a strict answer contract.

System rules:
- Use only the supplied source passages.
- Cite the source document and location for every material claim.
- If evidence is incomplete or conflicting, say so.
- Ask a clarifying question when the request is ambiguous.
- If no passage supports the answer, return NOT_FOUND.

Output:
- answer
- citations[]
- confidence_reason
- needs_human_review

Do not confuse a prompt instruction with enforcement. Validate the output shape, check that citations point to retrieved passages, and create tests where the correct answer is deliberately absent.

The retrieval debugging lab

When an answer is wrong, classify the failure before changing the prompt.

Keep a failure notebook. The fastest way to improve a RAG system is to turn each failure into a regression test.

Build a small evaluation lab

Use a test set that includes:

• Questions with one obvious source
• Questions requiring two sources
• Questions whose answer is absent
• Questions with conflicting document versions
• Questions containing synonyms or vague language
• Questions that attempt to override the system rules

Track retrieval and answer quality separately.

A reproducible build sequence

1. Create ten safe documents and record owner, version, and access group.
2. Write ten questions with expected sources and three intentional NOT_FOUND cases.
3. Extract text and manually inspect difficult files.
4. Index heading-aware chunks with source metadata.
5. Implement authorization filters before search.
6. Return retrieved chunks in a debug view for development.
7. Generate answers with citations and an abstention value.
8. Validate citations against retrieved source spans.
9. Run the evaluation lab and save results.
10. Only then add reranking, hybrid search, or a larger model.

This sequence keeps the system explainable. At every stage you can answer what changed and which failure class it affects.

What changes before production

A learning project can use a small index and a few documents. A real system needs lifecycle decisions.

• Document replacement must remove stale chunks.
• Access rules must be tested for every workspace and role.
• Deletion must cover indexes, caches, logs, and backups where applicable.
• Provider retention and zero-data-retention options require explicit review.
• Prompt injection must be treated as untrusted document content.
• Costs need a limit per user, workspace, and request.
• Slow retrieval or provider failures need a user-visible fallback.

Do not claim “private” because the model cannot browse the public internet. Privacy depends on your storage, authorization, provider settings, logs, and deletion behavior.

False fixes that waste time

• Use a larger model: does not repair lost or unauthorized source text.
• Increase top-k forever: adds noise and makes grounding harder.
• Make chunks huge: hides relevant passages inside irrelevant context.
• Put authorization in the prompt: prompts are not access controls.
• Add more documents: expands the search space before the pipeline works.
• Trust citations automatically: a citation can point to a real page without supporting the claim.

Fix the earliest broken gate first. Better generation cannot compensate for bad evidence.

Frequently asked questions

Is RAG the same as fine-tuning?

No. RAG retrieves current context at request time; fine-tuning changes model behavior through training. Choose based on whether the problem is knowledge access, behavior, format, or a combination.

Do I need a vector database?

No. Start with a small local index or database if that makes the concepts easier to see. Adopt a specialized store when scale, filtering, or operational needs justify it.

How do I stop unsupported answers?

Retrieve and authorize carefully, require citations, test absent evidence, validate claims where possible, and provide a clear abstention path.

Can RAG handle tables?

It can, but tables need deliberate extraction and chunking. Preserve headers, row relationships, page metadata, and a test set containing table questions.

What is the first RAG project a beginner should build?

Use a small collection of safe documents and a narrow question set. The goal is not broad knowledge; it is an observable pipeline you can debug.

Build an evidence pipeline, not a magic answer box

The useful lesson of RAG is not that a model can read documents. It is that a product can connect changing knowledge to a user task while making evidence inspectable.

Start with questions. Preserve the source. Filter access before retrieval. Debug each gate. Measure retrieval separately from generation. Make unsupported answers safe. Then expand only when the evidence says the system is ready.