Large language models write fluent, confident answers. Left alone with enterprise knowledge, they will also write answers that have no basis in your actual policies, contracts, or technical documentation. In regulated environments, a single fabricated answer can create real legal and operational risk.
Retrieval-Augmented Generation (RAG) reduces that risk by grounding answers in your own documents. Done well, every answer points to the passages it relied on, and the system declines to answer when the evidence is not there. Done poorly, it still invents, just with footnotes. The difference comes from architecture decisions, not from model selection alone.
Start with ingestion, not the model
Most RAG failures begin before a single question is asked. If documents are split into fixed-size token windows with no regard for meaning, sentences are cut mid-thought, tables lose their headers, and a clause is separated from the section that qualifies it. Retrieval then returns fragments that look relevant but mislead.
We recommend structure-aware chunking:
- Respect document structure. Split on headings, sections, and list boundaries. Keep a heading path (for example, "HR Policy > Leave > Sick Leave") attached to each chunk as context.
- Handle tables and forms deliberately. Convert tables to a consistent text or Markdown representation and keep the header row with every chunk that contains rows.
- Keep chunks reasonably sized with modest overlap, and tune the size against your own evaluation set rather than a default.
- Record metadata at ingestion: source document ID, version, page or section, language, effective date, and access policy.
For Arabic and bilingual content, normalize text consistently (for example, letter variants and diacritics) and confirm that your embedding model actually performs well on Arabic. Test it; do not assume.
Retrieval must respect permissions
Enterprise knowledge is not uniformly accessible. An assistant that retrieves HR investigation notes for a customer service agent has created a data leak, even if the final answer looks harmless.
The rule we design for is simple: filter by permission before anything reaches the model. Every chunk inherits the access policy of its source document, and the retrieval query includes the user's identity and groups as a filter in the vector or hybrid search. Filtering after generation is too late, because the model has already seen the content.
Practical points:
- Sync permissions from the system of record (SharePoint, a document management system, or your own application) on a schedule, and re-index when they change.
- Combine semantic search with keyword search (hybrid retrieval). Product codes, contract numbers, and names are often matched better by keywords.
- Add a re-ranking step on the top results to improve precision before passing context to the model.
Generate with constraints and citations
Once the right passages are retrieved, the generation step should be constrained:
- Instruct the model to answer only from the provided context and to say clearly when the context does not contain the answer.
- Require citations that reference chunk IDs, and render them in the interface as links to the exact document and section.
- Validate citations after generation: if the answer cites a source that was not retrieved, or cites nothing at all, treat it as a failed answer.
- Prefer quoting short passages for high-stakes questions, such as policy terms or medication dosages, so the user can verify the exact wording.
Citations change user behavior. When people can click through to the source, they check it, and trust is earned rather than assumed.
Evaluate before and after launch
RAG quality cannot be judged by a handful of demo questions. We recommend building an evaluation set with the people who will use the system: real questions, the documents that should answer them, and acceptable answers. Then measure each part separately:
- Retrieval quality: did the right passages appear in the top results?
- Faithfulness: is every claim in the answer supported by the cited text?
- Refusal behavior: does the system decline when the answer is not in the corpus?
Automate what you can: a separate model can score faithfulness across hundreds of answers, but keep people reviewing a sample so the scorer itself is checked. Run this evaluation on every change to chunking, embeddings, prompts, or models, just as you would run tests on code, and keep running it in production on real queries, because document collections and questions drift over time.
Audit everything
In regulated settings, the question after an incident is always the same: what did the system show, to whom, and why? Log each query, the user and their permissions at that moment, the retrieved chunk IDs and document versions, the model and prompt version, and the final answer with its citations. Store these logs with retention rules that match your policies, and protect them as sensitive data in their own right.
Where the model runs
For many organizations, the safest option is to keep both the index and the model inside their own perimeter. We often deploy RAG with open-weight models hosted on-premise or in a private cloud, so no document is sent to third-party model endpoints. Where policy allows a managed cloud model, the same architecture applies: permission filtering, citations, evaluation, and audit do not depend on where inference happens.
RAG is not a demo. It is an architecture that has to survive security review, compliance audit, and daily production use. That is the bar we hold our own deployments to: structure at ingestion, permissions at retrieval, citations at generation, and an evaluation set that runs on every change. Designed that way from the start, it becomes a system people can rely on.
