Beyond Vector Search: Engineering Reliable Enterprise RAG with GraphRAG and Contextual Retrieval

Executive summary

Vector search is a good place to start with retrieval-augmented generation (RAG), and for narrow use cases it may be all you need. It gets shaky when questions hinge on exact identifiers, on context a chunk boundary cut off, on relationships spread across documents, or on knowing which version of a document is the one in force.

This piece walks through the techniques that address those failures (hybrid search, reranking, Contextual Retrieval, graph-based retrieval and bounded agents), what each one costs, and how to decide which ones a use case has actually earned. Permissions, evaluation and ownership are treated as part of the architecture, not as an afterthought. No design guarantees correct answers. The realistic goal is evidence that is permitted, inspectable and measurable, with a clear way to say "I don't know" or hand the question to a person.

Where enterprise RAG usually breaks

Enterprises rarely lack documents. What they lack is a dependable way to tell which ones belong together and which one is current. Contracts live in one system, policies in another, tickets in a third, and the person who knows the answer sits in a fourth.

Put an assistant across that mess and answer quality comes down to a step that never shows up in a demo: what got retrieved, and what didn't.

RAG itself is well established. Lewis et al. (2020) paired a language model with a dense vector index so that answers could draw on retrieved passages. The idea is sound. What changes is the corpus. A pipeline that looks great on a tidy demo set behaves differently on real documents, and when it fails, it tends to look like the model's fault.

Usually it isn't. If the model was handed incomplete, stale or irrelevant evidence, a fluent wrong answer is exactly what you should expect. NIST's Generative AI Profile calls this confabulation, confidently stated but false content, and notes that people colloquially call it hallucination. A better model can make better use of evidence. It can't use a document that was never retrieved. So the first habit worth building is diagnostic: look at what was retrieved before you judge what was generated.

Where basic vector search falls short

To be fair to vector search, the technique isn't the problem. The naive pipeline around it is: cut documents into fixed-size chunks, embed them, return the top few by similarity, and stop there. Five things tend to go wrong as the corpus and the questions grow.

Similarity isn't relevance. A passage can sit close to a question in meaning without answering it. Termination clauses from dozens of agreements look alike, and two entities with the same name are hard to pull apart in embedding space.

Exact terms still matter. Part numbers, clause IDs and internal acronyms are often better served by lexical matching. BM25 has been a durable ranking function for decades (Robertson and Zaragoza, 2009). Dense retrieval can beat it when trained for a task: Karpukhin et al. (2020) reported gains of 9 to 19 points in top-20 accuracy on several open-domain question-answering datasets. Yet BEIR (Thakur et al., 2021), which tests zero-shot generalization across many domains, found BM25 a robust baseline and dense and sparse models often underperforming other approaches. So the honest answer depends on your domain. Measure on yours.

Chunks lose their surroundings. "The notice period is 30 days" tells you little without the agreement, the party and whatever exception comes next.

Similarity can't see authority or time. A superseded procedure and its replacement can be nearly identical in meaning, and the score has no idea which one is in force.

And piling on more context isn't a fix. Liu et al. (2023) found that models used relevant information less reliably when it sat in the middle of a long input. Better selection beats a bigger window.

The building blocks, and what each one costs

Hybrid retrieval and rank fusion

Hybrid retrieval runs lexical and semantic search side by side and merges the results. Reciprocal Rank Fusion (Cormack, Clarke and Büttcher, 2009) is a simple, unsupervised way to combine rankings, and in the authors' experiments it beat individual systems and the Condorcet Fuse baseline.

I'd treat it as the default first upgrade, because it fixes exact-term misses without any new modeling. The price is a second index and fusion behavior you'll need to tune against your own questions.

Reranking

A reranker rescores a candidate set with a heavier model, often a cross-encoder, so the best evidence rises to the top. Nogueira and Cho (2019) showed BERT-based passage reranking reaching state-of-the-art results on the benchmarks they tested, and BEIR found that re-ranking and late-interaction models had the best average zero-shot performance, at high computational cost.

That cost is latency and spend on every query. Reranking pays off when the right evidence is already in your candidates but sitting too low. If it isn't in the candidates at all, a reranker can't help.

Contextual Retrieval

Contextual Retrieval, described by Anthropic in September 2024, asks a language model to write a short description that situates each chunk within its document, then prepends it before embedding and BM25 indexing.

Anthropic reported that, averaged across the domains it tested, this cut the top-20-chunk retrieval failure rate by 35 percent with contextual embeddings alone (5.7 to 3.7 percent), by 49 percent when combined with contextual BM25 (to 2.9 percent), and by 67 percent once a reranker was added (to 1.9 percent). Read those as vendor-reported results from specific datasets, embedding models and one specific reranker. They show the technique can help under those conditions. They are not independent evidence, and they aren't a production promise.

The costs are real. Indexing needs a model call per chunk. Generated context can be wrong, and once it's in the index it stays there. Changed documents need re-contextualizing. Anthropic itself points out that for knowledge bases under roughly 200,000 tokens, passing the whole thing in the prompt may be simpler than building retrieval at all. And cheaper tools usually come first: preserving document structure, attaching metadata such as title, type, effective date and owner, and parent-child retrieval.

Graph-based retrieval

"GraphRAG" gets used in two senses. Broadly, it means retrieval that leans on a graph of entities and relationships. Specifically, it's the method from Edge et al. (2024) at Microsoft Research, which has a language model extract an entity graph, groups it into communities, summarizes them, and answers corpus-wide questions from those summaries. Microsoft's documentation describes several query modes: local search for questions about specific entities, global search for questions about the dataset as a whole (which it describes as resource-intensive), a DRIFT mode that blends the two, and a basic vector mode for comparison.

Graph methods suit relationship-heavy questions: what depends on a system being retired, which agreements link to which amendments and subsidiaries, which themes recur across thousands of reports.

The evidence deserves a careful read. The authors evaluated on two corpora of about one million tokens each, podcast transcripts and news articles. Their sensemaking questions had no gold-standard answers, so they used an LLM evaluator for head-to-head comparisons on criteria such as comprehensiveness and diversity. They reported gains over a conventional RAG baseline on those measures, and noted that generalization to other domains, and a comparison of fabrication rates, remain open. That's a promising research result. It isn't proof a graph will improve accuracy on your documents.

The costs are modeling and upkeep. Extraction errors become retrieval errors. Someone has to own the schema and entity resolution. The graph has to stay current as documents change, and evaluation gets harder. A knowledge graph also doesn't solve multi-hop reasoning by itself: it has to represent the right relationships, and the question still has to be routed and answered correctly. Microsoft's own documentation recommends tuning prompts rather than expecting good results out of the box.

My bias is to resist the graph until your question set shows failures that hybrid search and reranking can't fix.

Bounded agentic retrieval

Agentic retrieval lets a model plan searches, reformulate queries, follow relationships and decide whether it has enough evidence. The research lineage includes ReAct (Yao et al., 2022), which interleaves reasoning with actions, and Self-RAG (Asai et al., 2023), which trains a model to decide when to retrieve and to critique what it retrieved.

It's a way to handle questions a single pass can't, not a more accurate default. Each step adds latency and tokens, behavior can vary from run to run, loops are possible, and every tool call is another place something can fail. So it needs hard limits: a maximum number of steps, timeouts, a token budget, an allow-list of tools, clear stop conditions and a fallback, including abstaining. Every tool call should run with the requesting user's permissions and be logged.

Security and governance belong inside the architecture

Controls bolted on after the pipeline exists tend to be weak. These belong in the design from the start.

Authorize before content reaches the model

If retrieval ignores who is asking, the system can surface material the user shouldn't see, and telling the model to hide it is a thin control. OWASP's guidance on vector and embedding weaknesses describes cross-group leakage in shared stores and recommends permission-aware retrieval and strict partitioning. Permissions should be checked against current entitlements at query time, in retrieval and in every tool call.

Tenant isolation and caches

Separate tenants by index or by enforced filters, and test the separation instead of assuming it. Watch the caches too. A response or retrieval cache that ignores entitlements can serve one user's content to another.

Retrieved content is untrusted input

Greshake et al. (2023) showed that applications which pull in external data blur the line between data and instructions, so hidden instructions in a document can steer behavior. OWASP puts prompt injection first in its LLM Top 10 and says it's unclear whether fool-proof prevention exists. Mitigations reduce impact rather than eliminate it: label external content as untrusted, limit which tools the model can call, validate outputs, and require a person's approval for consequential actions.

Source integrity, freshness and deletion

Poisoned or manipulated documents can enter a knowledge base, so validate inputs and keep provenance and lineage for every chunk. Updates and deletions have to reach every index, graph and cache. A deleted document that's still retrievable is both an accuracy problem and a governance problem.

Logs and prompts hold sensitive data

Retrieval logs, prompts and answers can contain the very information you're protecting. They need redaction, access control and retention rules of their own.

A citation is not correctness

A citation shows where a claim came from. The passage may be misread, outdated or wrong. Treat citations as an aid to review, not as proof.

None of this makes an organization compliant with the EU AI Act or any other regulation on its own. Map technical controls to your actual obligations with qualified advisers.

Evaluating retrieval and answers separately

Retrieval and generation fail for different reasons, so measure them separately. For retrieval, ask whether the needed evidence shows up and how high it ranks. For answers, ask whether they're correct against human-reviewed references, whether each citation really supports its claim, and whether the system declines when evidence is thin. Also test access control (does anyone ever receive content they shouldn't?) and track latency, cost and reliability per question type at realistic load.

Frameworks such as Ragas (Es et al., 2023) offer reference-free measures of retrieval relevance, faithfulness to context and answer quality. They're useful for scale and regression testing. But an LLM-based judge is a proxy, not ground truth. Even the GraphRAG authors checked their LLM evaluator against a claim-based measure and reported only moderate agreement. Validate automated judgments against people, especially on high-risk questions.

The test set matters more than the tooling. Include real business questions, hard cases (superseded documents, ambiguous entities, missing evidence), adversarial inputs, and reference answers written or reviewed by domain experts. Run it as a regression suite whenever the model, prompts, chunking or index change, and feed production incidents back into it.

What it takes to make RAG production-ready

Choosing techniques is the smaller part of the job. The judgment that matters looks more like this.

Diagnose retrieval separately from generation. For every failure, ask whether the right evidence was retrieved. If it wasn't, a better model won't help.

Measure a baseline before adding complexity. Without one on your own questions, you can't tell whether a new layer helped or just added cost.

Change one thing at a time. Hybrid retrieval, then contextual enrichment, then reranking, each compared with the step before. Reach for a graph or an agent only when measured failures remain.

Weigh quality against latency, infrastructure and upkeep. A small gain that doubles indexing cost, or needs a graph team to maintain, may not be worth it. A large gain on a high-stakes question might be.

Treat access control as a retrieval property, and test it with realistic users and groups, not only at the interface.

Test against what people will actually ask: real questions, edge cases and adversarial inputs, reviewed by people who know the domain.

Name owners. Someone owns data quality and metadata, someone owns evaluation, someone owns security, and someone owns production operations and incident response. Without owners, test sets go stale and metadata rots.

Choosing between approaches

These techniques complement each other. The real question is which failure you can measure and what you're willing to operate.

An illustrative scenario

This is a hypothetical example, not a real organization or result.

A legal team at a multinational manufacturer asks an internal assistant: "Which supplier agreements allow termination for convenience on short notice, and which of those were amended after the new regional regulation took effect?"

A basic pipeline returns termination clauses from many agreements, some irrelevant, and misses the amendments, because their wording isn't close to the question.

A layered design handles it differently. Lexical matching catches the exact clause terminology. Chunks carry the agreement name, counterparty and effective date, so they're unambiguous. A router recognizes a relationship-heavy question and uses a graph linking agreements to amendments, counterparties and subsidiaries. A bounded agent takes a few steps to compare amendment dates with the regulation date. Retrieval only considers agreements the requester may see. Where two agreements lack effective-date metadata, the system says so instead of guessing. And because termination is a consequential decision, the answer goes to a lawyer for review with its sources attached.

None of that guarantees correctness. It makes the evidence inspectable and the gaps explicit.

A 30-60-90-day evaluation roadmap

Set acceptance criteria from the use case, its risk level and the measured baseline. This plan deliberately avoids universal thresholds.

Days 1 to 30: scope and baseline

• Pick one use case with a named accountable owner and a risk classification.
• Document data sources, access requirements and who may see what.
• Collect representative questions from users, including hard ones, and build a reference set with human-reviewed answers and supporting sources.
• Measure the current baseline (plain vector search, or the existing process) for retrieval, answers and cost.
• Agree on success criteria and on what a failure would cost.

Days 31 to 60: controlled comparison

• Test candidate strategies one change at a time against the baseline.
• Measure retrieval quality, answer quality, citation support, abstention, latency and cost for each variant.
• Run permission, tenant-isolation, cache and prompt-injection tests with realistic users and adversarial inputs.
• Bring in graph or agentic components only where a measured failure justifies them.

Days 61 to 90: production readiness and decision

• Review governance: access control, provenance, retention, deletion propagation and audit trail.
• Confirm operational ownership, monitoring, incident response and regression testing.
• Estimate running and maintenance cost for each viable architecture.
• Make a go, no-go or narrowed-scope decision against the criteria set on day one, with executive review.

Conclusion

The useful shift isn't from vector search to something that replaces it. It's from treating retrieval as an implementation detail to treating it as a governed system with measurable quality. Vector search stays a sound component. Hybrid retrieval, reranking, contextual enrichment, graph methods and bounded agents each answer specific failures, and each has a price.

Start with a baseline. Add only what your measurements justify. Enforce permissions throughout, and keep a person accountable for consequential decisions. That won't deliver zero hallucinations. It can make unsupported answers rarer, easier to catch and easier to explain, which is a more honest target.

If you're evaluating an enterprise RAG system and want an independent view of its retrieval architecture, permissions model or evaluation plan, I'm happy to talk it through.

FAQ

Related reading on this site

• Secure enterprise RAG and knowledge intelligence (case study)
• Securing agentic AI workflows and multi-agent systems
• Why traditional security tools fall short against autonomous AI agents
• The demo-to-production gap in AI agents

References

Peer-reviewed and preprint research

• Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401.
• Robertson, S., and Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 4(1-2).
• Cormack, G. V., Clarke, C. L. A., and Büttcher, S. (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods. SIGIR 2009.
• Nogueira, R., and Cho, K. (2019). Passage Re-ranking with BERT. arXiv:1901.04085.
• Karpukhin, V., Oğuz, B., Min, S., et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. arXiv:2004.04906.
• Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., and Gurevych, I. (2021). BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. arXiv:2104.08663.
• Liu, N. F., Lin, K., Hewitt, J., et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172.
• Edge, D., Trinh, H., Cheng, N., et al. (2024). From Local to Global: A Graph RAG Approach to Query-Focused Summarization. arXiv:2404.16130.
• Yao, S., Zhao, J., Yu, D., et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629.
• Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. arXiv:2310.11511.
• Es, S., James, J., Espinosa-Anke, L., and Schockaert, S. (2023). Ragas: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217.
• Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., and Fritz, M. (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173.

Vendor-reported results and documentation

• Anthropic (19 September 2024). Introducing Contextual Retrieval. Vendor-reported benchmark results.
• Microsoft. GraphRAG documentation and query overview.

Official guidance

• NIST (July 2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1).
• OWASP GenAI Security Project. LLM08:2025 Vector and Embedding Weaknesses.
• OWASP GenAI Security Project. LLM01:2025 Prompt Injection.