AI Architecture · Insights

Disaggregated Inference Architecture and Lakehouse RAG: A Reference Design for Scalable Enterprise AI

How a governed lakehouse and a disaggregated GPU fleet fit together, with honest limits on when each is worth its complexity.

By: Asif Ali, Principal AI & Enterprise Architect
Published: November 2026

A RAG system is first judged on whether the answer looks right. It is later judged on whether it can be afforded, secured and kept fast when several hundred people use it at once. The second set of questions is architectural, and it has two halves that are rarely discussed together. One half is the data side: how knowledge travels from operational systems into something a retrieval layer can trust. The other is the inference side: how a GPU fleet turns a long, retrieved prompt into tokens without letting one workload starve another.

This article treats them as one system, for teams building scalable AI infrastructure that has to survive real tenants and real traffic. It explains disaggregated inference architecture, where the two phases of an LLM request run on separate pools, and lakehouse RAG, where an open table format sits beneath the indexes as the governed source of truth. Then it looks at the places where each layer pushes on the other: retrieval latency, context length, KV cache pressure and tail latency. It is a reference design, not a standard. Every number in the worked arithmetic is an assumption, not a benchmark, and I say so where it appears.

The short version: definitions and differences

Disaggregated inference architecture. Disaggregated inference architecture runs the two phases of LLM inference, prefill (processing the prompt) and decode (generating tokens), on separate, independently scaled pools of GPUs, moving the KV cache between them. Each phase can use different parallelism settings, and prefill work no longer interrupts token generation.

Lakehouse RAG. Lakehouse RAG builds retrieval indexes from data held in open table formats such as Apache Iceberg, Delta Lake or Apache Hudi, so documents, versions, lineage and access rules live in governed tables, and the indexes are rebuildable derivatives rather than the system of record.

Scroll sideways to see all columns
DimensionTraditional (colocated) inferenceDisaggregated inference
Where the phases runPrefill and decode share the same GPUs and often the same batchesSeparate pools, each sized and tuned for its own phase
InterferencePrefill jobs can delay decoding of other requestsLargely removed, at the price of a transfer step
Parallelism and hardware tuningOne configuration serves both phasesPer-phase configuration
KV cacheStays on the GPU that computed itMust move between pools, or be shared through a cache tier
Dependency on the networkLowHigh: transfer speed and reliability become part of latency
Operational complexityLowerHigher: routing, two pools, transfer layer, more failure modes
Typical fitShort prompts, modest scale, throughput-first workloadsLong prompts, strict TTFT and inter-token targets, large shared fleets

Why colocated LLM inference hits limits at scale

An LLM request has two phases with different personalities. Prefill reads the whole prompt in one pass and builds the key-value (KV) cache. It is dense computation, and its cost grows with prompt length. Decode then produces the answer one token at a time. Each step is small in arithmetic terms but has to read the model weights and the growing KV cache again, so it is memory-intensive and tends to leave compute idle. The Splitwise paper describes exactly this split, a compute-intensive prompt phase and a memory-intensive token-generation phase with different latency, throughput, memory and power profiles [2].

Run both phases on the same GPUs and they interfere. A long prefill that arrives mid-batch delays the decode steps of requests already streaming, which users experience as a stutter in the output. DistServe, which proposed separating the phases, ties each phase to its own latency target: how soon the first token appears (TTFT) for prefill, and how quickly each later token follows (TPOT) for decoding. A colocated system has to favor one target over the other, or over-provision to satisfy both [1]. vLLM's own documentation gives the same two reasons for its disaggregated prefilling feature: tune TTFT and inter-token latency separately, and control tail inter-token latency, noting that chunked prefill can achieve the second goal but that choosing a good chunk size is hard in practice [5].

RAG makes this sharper. A RAG prompt is long by construction, because retrieved passages are pasted into it. Prefill is therefore a large share of the work for every request, and the KV cache each request holds during decoding is large too. An enterprise used to 500-token chat prompts can find itself serving 6,000-token prompts to the same hardware.

How prefill decode disaggregation works in practice

Mechanically the idea is simple. A router sends the request to a prefill worker, which builds the KV cache and produces the first token. The cache then moves to a decode worker, which streams the rest of the answer. Everything difficult lives in that middle step. The ecosystem has several ways to do the transfer: vLLM documents connectors that include NIXL and Mooncake [5], NVIDIA's NIXL library is aimed at point-to-point transfers between inference nodes across memory and storage types [10], and SGLang's PD disaggregation documentation names Mooncake and NIXL as transfer engines [11]. NVIDIA Dynamo and TensorRT-LLM offer disaggregated serving too, though TensorRT-LLM's documentation labels its feature beta or prototype, so check the status of the version you run [9][12].

Once the phases are separate, they no longer have to share a configuration. Prefill can use settings that favor compute throughput on long prompts, and decode can use settings that favor memory capacity and bandwidth. vLLM's documentation points at exactly this freedom, with different tensor and pipeline parallel strategies per phase [5]. Hardware can differ as well: Splitwise argued that token generation does not need the newest compute and can run on cheaper or lower-power hardware [2].

The router is the part I would worry about first. In a colocated fleet it mostly balances load. Here it has to pick a prefill worker with room, pick a decode worker with enough free KV memory, and, where it can, favor a worker that already holds the relevant prefix. Dynamo documents KV-aware routing and an SLA-based planner for scaling [9]. llm-d describes prefix-cache-aware routing, tiered prefix caches, prefill/decode disaggregation and flow control as paths on Kubernetes [13], and the Gateway API Inference Extension, an official Kubernetes project, routes traffic to self-hosted generative models through an inference gateway [14]. Mooncake goes furthest, combining a KV-cache-centric scheduler, a cache pool built from the CPU, DRAM and SSD of the GPU cluster, and an early-rejection policy for overload [3]. LMCache frames KV cache as something to store, reuse across serving engines and monitor, instead of a transient by-product [15].

The network deserves its own arithmetic, because it is now part of your latency. Assume a model with 32 layers, 8 KV heads of dimension 128 and 16-bit values. The cache holds two tensors, keys and values, per layer per head, so one token costs 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes, or 128 KiB. A 6,500-token RAG prompt therefore carries about 0.79 GiB. Moving 1 GiB takes about 86 ms at a theoretical 100 Gbit/s and about 21 ms at 400 Gbit/s, both line-rate lower bounds that ignore protocol overhead. Now assume 200 such requests per second. The aggregate is roughly 1.4 Tbit/s, more than any single link carries. That does not sink the idea. It means transfer bandwidth has to be designed, measured and monitored like any other tier.

Separate pools also scale on separate signals: prefill on queue depth and prompt-token rate, decode on KV memory use and concurrent streams. Kubernetes covers the GPU side through device plugins and, since Dynamic Resource Allocation became stable in 1.35, a richer device-request model [16][29]. Multi-model and multi-tenant serving then add per-tenant concurrency and token budgets at the gateway, plus dedicated pools for tenants whose isolation requirements justify them. I covered that cell-based approach in the cellular AI architecture article.

When it pays, and when it does not

Be careful with the sales pitch for this pattern. vLLM's documentation is blunt: disaggregated prefill does not improve throughput [5]. What it gives you is control: of TTFT and inter-token latency independently, and of tail latency under mixed workloads. Whether that is worth a transfer layer, a more complex router and two pools to operate depends on facts you can measure.

It tends to make sense when prompts are long and variable, latency targets are strict on both TTFT and streaming smoothness, the fleet is large enough that two right-sized pools beat one compromise configuration, and the interconnect is fast and well understood. Mooncake's reported gains came from long-context workloads on a large production service [3], so be careful extrapolating them to smaller deployments.

It tends to add complexity without paying for itself when prompts are short, traffic is small, a single replica or a few colocated replicas already meet the latency objectives, or the team cannot yet operate an inference engine well. In that case, prefix caching and a well-chosen chunked-prefill setting are the sensible first moves [5][8]. Disaggregate when the measurements show interference or when pool sizing is clearly fighting you, not because the architecture diagram looks better.

Lakehouse RAG: an enterprise RAG architecture built on governed data

A vector database stores embeddings and answers nearest-neighbor queries. That is useful and incomplete. An enterprise knowledge architecture has to answer other questions as well: where did this passage come from, which version of the source document was it, who is allowed to see it, when was it last refreshed, and can we rebuild the index after we change the embedding model. A vector index alone cannot answer those, because it is a derivative of the data, not the data.

Start with where the truth lives. In a lakehouse design, raw and parsed documents, chunk tables, embedding metadata, access attributes and evaluation results sit in governed tables on object storage, in an open table format. The three common formats make broadly comparable promises. Iceberg supports in-place evolution of schemas and partition layouts, plus branching and tagging of snapshots [19][20]. Delta Lake offers time travel and a change data feed [21]. Hudi keeps a commit timeline and supports incremental and change-data queries [22]. I would not choose among them on feature lists alone. The engines and catalog your organization already runs, and how well the format supports the incremental reads your indexing pipeline needs, matter more.

Data reaches the lakehouse from operational databases, SaaS APIs, document stores and event streams. Change data capture and streams give freshness, scheduled batch loads give completeness and backfill, and both land in the same raw layer with source identifiers and timestamps intact. If you already run Kafka and Flink, my earlier piece on event-driven AI architecture covers the idempotency and replay questions that come with this.

From there, parsing, cleaning, chunking and embedding are just transformations over tables, so they inherit versioning and lineage. Each chunk carries its source document identifier, version, position, classification and the access attributes that applied when it was created. Embedding becomes a job that reads chunk rows and writes vectors, which turns an embedding model upgrade into a reindexing exercise you can run in parallel, compare and cut over, instead of a mystery.

Retrieval has to be hybrid. Dense vector search finds semantic matches, keyword search finds the exact identifiers, product codes and names that embeddings blur, and metadata filters enforce tenant, region, recency and permissions. Merging the result lists is a known problem: Elasticsearch documents reciprocal rank fusion for combining multiple ranked lists [23], and OpenSearch documents hybrid search with pre-filtering and post-filtering options [24]. A reranker then reorders the top candidates with a more expensive model, and a context assembler decides what goes into the prompt: how many passages, in what order, with which citations, within a token budget.

Freshness is a property of the pipeline, so measure it as the lag between a change at the source and its availability in the index, and use incremental reads from the table format rather than re-embedding everything. Deletions need extra care. A document removed or restricted at the source has to disappear from retrieval within a stated time, which means tombstones must reach the index.

Because every chunk traces back to a table row and a source, audit questions and deletion requests have an answer. Access attributes travel with the chunk so retrieval can filter on the caller's identity. For multimodal sources such as scanned documents, images or audio, the same pattern applies: extract text or embeddings, keep the original object and its metadata in the lakehouse, and index the derived representations.

Connecting distributed LLM inference and RAG

This is the part that is easiest to leave out of an architecture diagram. The two halves may be owned by different teams and watched on different dashboards, so nobody owns the seam. The request path runs through eight stages: ingestion and lakehouse storage, transformation, embedding and indexing, retrieval and ranking, context preparation, disaggregated inference, response streaming, and feedback with evaluation. The first three run mostly offline. The remaining five sit on the user's clock, and several variables cross between them.

Context tokens are the currency. Retrieval decides how many tokens enter the prompt, and every token costs prefill compute, KV memory for the entire decode duration, and transfer bandwidth in a disaggregated fleet. Doubling top-k does not just double retrieval work. It lengthens the prefill, enlarges the cache, shrinks the batch the decode pool can hold and raises cost per query. Reranking to a smaller, better context is one of the first levers to test, because it reduces prefill work and KV memory together.

Latency is a budget, not a number. What the user perceives as TTFT includes retrieval, reranking, queueing and prefill before the first token appears. Give each stage an explicit budget and measure it separately, because a slow retriever and a congested prefill queue look identical from the outside. Pass the remaining deadline downstream so a stage can refuse work it cannot finish.

Cache behavior follows from how you assemble the prompt. vLLM's automatic prefix caching reuses the cache for requests that share a prompt prefix [8]. In RAG the shared part is the system prompt, tenant instructions and tool definitions, while the retrieved passages change with every query. Put the stable material first and the variable material last, or the hit rate collapses for no good reason. Expect most of the benefit to come from that stable prefix, and measure instead of assuming.

Memory is the other constraint. Decode pools tend to run out of room for KV caches before they run out of arithmetic, and the cache is a memory-management problem in its own right: the PagedAttention paper behind vLLM set out to reduce the waste that fragmentation and duplication cause in KV cache memory [4]. Long RAG contexts cut concurrency per GPU directly, which is one place where disaggregation helps economically. You can size and configure the decode pool for memory and the prefill pool for compute, instead of compromising on one shape.

Pressure also has to travel upstream. If the prefill queue is deep, there is no point retrieving and reranking for requests that will wait too long or be rejected, so admission control belongs at the gateway, fed by signals from the inference pools. Mooncake's early-rejection idea reflects the same logic: under overload it is better to reject early than to spend resources on work that cannot meet its deadline [3]. Cancellation matters for the same reason. When a client disconnects, cancel the decode stream so its KV cache is released and capacity returns to other requests.

Finally, close the loop through the lakehouse. Log the query, the retrieved chunk identifiers and versions, the assembled context size and the response, and write them to tables. Evaluations, human feedback and retrieval quality metrics then join cleanly to the data and model versions that produced them, and a regression test on an embedding upgrade becomes an ordinary query.

None of this requires everything to be distributed. Retrieval services can sit beside a modest colocated inference engine until measurements say otherwise. The decisions that deserve deliberate design are the budgets, the identity that follows a request through every stage, and the telemetry that lets you tell which layer is hurting.

Reference architecture for scalable AI infrastructure

Figure 1 shows a logical design for a platform serving multiple tenants, millions of documents, high-volume RAG traffic and several models. Thousands of concurrent users is a design target, not a benchmark. Solid arrows are data flow, dashed arrows are control and telemetry flow, and the two inference pools are drawn separately to make the KV transfer visible.

Scroll sideways on small screens to see the whole diagram
Governance and operationsDisaggregated inferenceRequest pathIndex buildLakehouse (governed system of record)IngestionEnterprise sourcesTraces, metrics and SLOsAudit log and evaluation tablesCost attributionModel registry and policyPrefill pool (GPU)KV transfer layer and cache tierDecode pool (GPU)API gateway, identity and tenant resolutionQuery service: rewrite, hybrid retrieval, ACL filterReranker and context assemblerKV-aware inference router and admission controlEmbedding workersVector and keyword indexesMetadata and permission storeRaw and parsed tablesChunk, version and ACL tablesCatalog, lineage and access policyCDC, connectors and batch jobsEvent backboneDatabases and SaaS APIsDocuments and object storesEvent streams
Figure 1. Reference architecture. Solid arrows are data flow and dashed arrows are control and telemetry flow.

Layers and responsibilities.

  • Sources and ingestion. Connectors and change data capture bring database and SaaS changes, a document pipeline brings files, and an event backbone carries streams. Everything lands raw, with provenance. Responsibility: completeness, ordering where needed, replay.
  • Lakehouse. Open-format tables on object storage hold raw, parsed and chunked data with versions and access attributes, managed through a catalog that records lineage and enforces policy. Responsibility: governance, reproducibility, the single source of truth.
  • Index build. Embedding workers, often on GPUs, read chunk tables incrementally and write vector and keyword indexes plus a metadata and permission store. Responsibility: freshness, rebuildability, isolation per tenant.
  • Request path. The gateway authenticates the caller and resolves the tenant. The query service rewrites the query, runs hybrid retrieval and filters by permissions before anything leaves the index. The reranker and context assembler shape the prompt to a token budget. Responsibility: relevance, authorization, latency budgets.
  • Inference router. A KV-aware router with admission control chooses workers, rejects early under overload and routes by tenant tier and model. Responsibility: protecting the pools and enforcing fairness.
  • Disaggregated inference. The prefill pool builds the cache, the transfer layer moves or shares it, and the decode pool streams tokens. A tiered cache can offload cold prefixes. Responsibility: meeting TTFT and inter-token objectives at an efficient cost.
  • Governance and operations. Telemetry, audit logs, evaluation tables, cost attribution and a model registry with policy close the loop.

The request lifecycle. The gateway authenticates the request and attaches tenant and user identity. The query service retrieves candidates through hybrid search with the permission filter applied inside the query. The reranker narrows the candidates and the assembler builds the prompt with stable content first. The router checks the deadline and pool capacity, then sends the request to a prefill worker. The cache moves to a decode worker, which streams tokens back through the gateway. Audit and evaluation records are written to the lakehouse. A client disconnect cancels the decode stream.

Failure scenarios.

Scroll sideways to see all columns
FailureDetectionArchitectural responseTrade-off
Prefill pool saturatedQueue depth and prefill wait timeAdmission control, priority classes, shed low-priority work, scale poolSome requests are rejected or delayed on purpose
Decode worker fails mid-streamStream errors, health checksRetry from the start on another worker if the response is idempotent, or surface an errorRepeated prefill cost and a visible interruption
KV transfer degradedTransfer latency and error ratesFall back to colocated serving on a subset of workersLower efficiency, simpler failure mode
Index lagging or inconsistentFreshness lag metric, version mismatchServe with a freshness warning, rebuild from the lakehouseStale answers for a bounded time
Embedding model upgrade fails evaluationOffline evaluation gateKeep the old index and model, cut over only after parityDual-index cost during migration
Region outageHealth probes, DNS and routingFail over to a second region with replicated tables and rebuilt or replicated indexesRecovery point and time depend on replication choices
Policy service unavailableGateway errorsFail closed for sensitive tenantsAvailability sacrificed for safety

Deployment on Kubernetes and GPUs. Run prefill and decode as separate deployments with their own node pools and autoscaling signals. Use GPU device plugins or Dynamic Resource Allocation, topology-aware placement so cache transfers stay on fast links, namespaces and network policy per tenant tier, and dedicated node pools for high-isolation tenants [16][17][18][29]. Keep the retrieval services on CPU nodes next to the indexes. Treat the router and the transfer layer as critical infrastructure with their own on-call ownership.

Cloud-agnostic considerations. Object storage, Kubernetes, open table formats and open inference engines are available across providers, so the design ports. The places where it does not port cleanly are the interconnect (the bandwidth and topology of GPU networking differs by provider and instance type), GPU availability and managed services for vector search. Abstract the storage and queue interfaces, but benchmark the transfer path on each target environment instead of assuming it behaves alike.

Sample stack, with alternatives.

Scroll sideways to see all columns
LayerExample optionsSelection criteria
Event backboneApache Kafka, Kafka-compatible platforms, cloud queuesThroughput, replay needs, operations skills, cost
Table formatApache Iceberg, Delta Lake, Apache HudiEngine and catalog compatibility, incremental read support, existing platform
ProcessingApache Flink, Apache SparkLatency target, state size, team skills
Search and vector indexesOpenSearch, Elasticsearch, Milvus, Qdrant, pgvectorHybrid search quality, filtering, scale, tenant isolation, operations
Embedding and rerankingOpen-weight or hosted modelsQuality on your data, latency, data residency, licensing
Inference enginevLLM, SGLang, TensorRT-LLMModel support, hardware, features, maturity of disaggregation for your version
Orchestration and routingNVIDIA Dynamo, llm-d, Gateway API Inference Extension, customKubernetes fit, KV-aware routing, scaling signals, operational maturity
KV transfer and cacheNIXL, Mooncake, LMCacheInterconnect, engine support, cache tiering needs
ClusterKubernetes with device plugins or DRAGPU scheduling needs, multi-tenancy, cloud portability
ObservabilityPrometheus-style metrics, OpenTelemetry traces, audit storeCross-stage tracing, tenant labels, retention

No stack above is universally right. The columns on the right are the real decision.

What breaks in production-grade RAG systems, and what to do about it

Scroll sideways to see all columns
ChallengeMitigationTrade-off
GPU scarcity and contentionPer-tenant budgets, priority classes, separate pools for embedding, reranking and generationLower peak flexibility for individual tenants
KV cache memory pressureSmaller, better contexts, paged cache, prefix reuse, a cache tier, admission controlAdded complexity and possible quality trade-offs from truncation
Network bottlenecks on cache transferTopology-aware placement, bandwidth monitoring, fallback to colocated servingPlacement constraints reduce scheduling freedom
Retrieval and tail latencyStage budgets, parallel retrieval branches, caching, timeouts, hedged requests used sparinglyExtra load and operational tuning
Hot partitions and skewKeys that spread load, rebalancing, tenant-aware shardingRebalancing cost and more complex routing
Embedding model upgradesParallel index, offline evaluation gate, staged cutoverTemporary double storage and compute
Index consistency and freshnessIncremental pipelines, version stamps, freshness SLOs, tombstonesPipeline complexity
Tenant noisy neighborsQuotas, concurrency limits, dedicated pools for high tiersLower utilization
Cross-tenant leakage and unauthorized retrievalPermission filters inside the retrieval query, per-tenant indexes or namespaces, identity propagation, tests that must fail closedCost of isolation and filter overhead
Prompt injection through retrieved documentsTreat retrieved text as untrusted data, constrain tools, validate outputs, limit what the model can do with itReduces impact but does not guarantee prevention
Model fallback and degradationFallback ladder: smaller model, cached answer, retrieval-only result, explicit errorQuality varies by rung
Observability across stagesEnd-to-end trace IDs, per-stage latency, tenant and model labelsTelemetry volume and cost
Cost attributionTag tokens, GPU time and retrieval cost by tenant and requestInstrumentation effort
Disaster recoveryReplicated lakehouse tables, rebuildable indexes, tested failoverRecovery time depends on index rebuild time

Performance, scalability and economics

GPU optimization for LLMs and enterprise AI cost optimization are one measurement problem seen from two ends, so start with what to measure. TTFT, time to first token, is the wait before any output appears. Time per output token (TPOT) and inter-token latency describe how smoothly the answer streams, and end-to-end latency covers the whole request. Track p50, p95 and p99 for each, because averages hide the users who are suffering. Add tokens per second, throughput per GPU, GPU utilization, KV cache usage and prefix cache hit rate. vLLM exposes metrics for most of these, including time to first token, time per output token, inter-token latency, end-to-end latency, KV cache usage, prefix cache hits and queries, and requests running and waiting [6]. On the retrieval side, measure retrieval and rerank latency and index freshness. For quality, evaluate answer faithfulness and retrieval relevance against a labeled set, release by release. For money, calculate cost per successful query and cost per million generated tokens.

The cost method fits in one sentence: GPU price per hour, divided by tokens generated per hour, multiplied by one million. Take an assumed $4.00 per GPU-hour and an assumed sustained 2,000 output tokens per second per GPU while still meeting the latency target. That is 7.2 million tokens per hour, or about $0.56 per million tokens. If real utilization is half of that, the cost doubles. This is only an illustration of the method. The number to trust is your own measured throughput at your own latency target, not one from a blog or a slide. Cost per successful query should add retrieval, reranking and embedding costs, and it should count failed or rejected requests in the cost but not in the denominator.

For capacity planning, Little's law gives a first estimate: concurrency equals arrival rate multiplied by time in the system. Suppose 1,000 concurrent users each send one query a minute, about 16.7 queries per second, with an assumed 8-second end-to-end latency. That is roughly 133 requests in flight. If each carries about 7,000 tokens of context at 128 KiB per token (the assumed model from earlier), the KV cache alone comes to roughly 114 GiB across the fleet at that instant, before model weights and headroom. That is why decode pools are sized for memory, and why context length is a capacity lever.

Load tests should resemble production. Replay a realistic mix of prompt lengths, output lengths, tenants and arrival patterns, including bursts and long-prompt outliers, with real retrieval in the loop instead of synthetic prompts alone. Ramp until the latency objectives break and record where, then test failure on purpose: kill a decode worker, saturate a pool, degrade the transfer link. vLLM includes benchmarking tooling [7], and AIPerf is an open-source tool for benchmarking generative AI serving [30]. Check any tool's workload against your own traffic, report percentiles instead of averages, and keep configuration and data versions with each run so results stay comparable.

Security, governance and compliance

Security belongs inside the architecture, not on top of it. Start from zero trust: no implicit trust between services, identity-based authorization, and encryption in transit and at rest. The caller's identity has to follow the request through the gateway, retrieval, reranking and inference, so every stage can apply the same decision.

Retrieval is where that decision matters most. Apply role-based and attribute-based rules inside the retrieval query itself, never after the fact, so the index is simply unable to return a chunk the caller may not see. For high-isolation tenants, use separate indexes or namespaces and, where justified, dedicated pools. Detect and redact PII during ingestion and again before logging, keep secrets in a secrets manager with short-lived credentials, and classify data when it enters the lakehouse so the label carries through to the index.

The OWASP GenAI Security Project's 2025 list includes prompt injection (LLM01), sensitive information disclosure (LLM02), data and model poisoning (LLM04), excessive agency (LLM06), vector and embedding weaknesses (LLM08) and unbounded consumption (LLM10) [25]. In a RAG system these map to concrete controls. Indirect prompt injection arrives inside documents, so retrieved text is untrusted input and the model's tools need least privilege. Poisoning arrives through ingestion, so source trust and write controls matter. Embedding weaknesses and leakage are addressed by permission-aware retrieval and tenant boundaries, and unbounded consumption by quotas and token budgets. Prevention is not guaranteed, so assume some injection will succeed and limit what it can reach.

Authenticate every inference endpoint, restrict network paths, and log who asked what, which chunks were retrieved by identifier and version, which model answered and what tools ran. Keep human oversight for decisions with material consequences.

On frameworks, the NIST AI Risk Management Framework and its Generative AI Profile give a structure for risk management [26][27], and NIST says a revision of AI RMF 1.0 is under way. The EU AI Act, Regulation (EU) 2024/1689, takes a risk-based approach, with obligations that vary by the risk level of the AI system [28]. An architecture can support these efforts with logging, lineage and access control. It does not by itself make an organization compliant.

The executive view of enterprise AI architecture

AI infrastructure scalability, in business terms, is whether growth in tenants and queries produces proportional cost and risk, or worse. Enterprise AI cost has several moving parts that scale differently: GPU time, retrieval and storage, embedding refreshes, and the people operating the stack. Prompts grow because retrieval adds context, utilization suffers when pools are sized for peaks, and nobody can attribute spend to the teams or tenants that caused it. Architecture shapes all three.

Disaggregation can improve resource utilization when mixed workloads force a colocated fleet to compromise between prefill and decode configurations, and when per-phase hardware choices become possible. It will not help a small, steady workload, and vLLM's documentation is explicit that it does not raise throughput on its own [5].

Governance deserves to come first for RAG. Adoption can stall when security and legal teams cannot see where answers come from or who can see what. A lakehouse with lineage and permission-aware retrieval gives them those answers early, instead of leaving them as late blockers.

On build versus buy, my rule of thumb is to buy what is a commodity for you and build what differentiates you or what you cannot hand off for compliance reasons. Managed inference and managed vector search reduce operations load. Self-hosted inference gives control over data, models and cost at scale, at the price of a skilled team. A hybrid is one option: managed components for experimentation, self-hosted for steady, high-volume or sensitive workloads. Revisit the decision when volume, regulation or model strategy changes.

Founders should settle a few things before scaling: unit economics per tenant and per query, an evaluation set before the first large customer, isolation tiers, and whether the demo's behavior survives real data, real permissions and real concurrency. Production readiness matters more than an impressive demo, because the demo is the best day your system will ever have.

A decision framework for executives. Score each question as clear, partially clear or unknown. Unknowns become the work plan.

Scroll sideways to see all columns
QuestionWhy it mattersEvidence to ask for
Do we know our prompt and output length distributions?They determine whether disaggregation helpsProduction or pilot traces by tenant
Are we missing TTFT or inter-token targets because of interference?The main technical reason to separate phasesp95 and p99 latency split by phase
Is our network fast and measured enough for KV transfer?Transfer cost can erase the benefitMeasured bandwidth and latency between pools
Can we attribute cost per tenant and per successful query?Without it, pricing and FinOps are guessesTagged token and GPU usage
Is retrieval permission-aware and tested?Cross-tenant leakage is a serious trust failureNegative authorization tests that fail closed
Can we rebuild every index from governed tables?Embedding upgrades and recovery depend on itA timed rebuild drill
Do we have an evaluation set tied to data and model versions?Quality needs a regression testVersioned evaluation results
Can we operate this?Complexity must match team capacityOn-call ownership and runbooks
What is the simplest design that meets the targets?Guards against over-engineeringA written comparison with a colocated baseline

Three hypothetical scenarios

These are hypothetical reference scenarios, not client engagements, and none describes a real organization.

Scroll sideways to see all columns
Financial services enterpriseLarge retail organizationFast-growing B2B SaaS company
Business problemEmployees need answers from sensitive internal policies, research and case recordsStaff and customers need answers across millions of product, supplier and operational recordsMany enterprise customers expect a knowledge assistant over their own data
Architectural requirementsStrict access control, audit, data residency, human oversightFreshness across fast-changing catalogs and inventory, high read volume, multilingual contentTenant isolation tiers, per-tenant cost attribution, rapid onboarding
Proposed solutionLakehouse with lineage and classification, permission-filtered retrieval, self-hosted inference in a controlled environment, colocated serving firstStreaming ingestion into the lakehouse, hybrid retrieval with metadata filters, incremental indexing, shared inference poolsPer-tenant namespaces or indexes, shared pools for standard tiers, dedicated pools for premium tenants, KV-aware routing as volume grows
Key technology choicesOpen table format, hybrid search with ACL filtering, private model hosting, mutual TLS, audit storeEvent streams and CDC, hybrid search, reranking, cache tier for popular queriesCell or tenant-tier architecture, quotas at the gateway, model routing by tier
Security and reliabilityFail closed on policy errors, PII redaction, tested negative cases, tested recoveryGraceful degradation to retrieval-only results, regional failoverLeakage tests per release, noisy-neighbor limits, tiered SLOs
Scaling strategyScale users first, add pools only when latency data demandsScale indexing and retrieval first, then inferenceScale by tenant cohorts, introduce disaggregation when prompts grow long and latency targets tighten
Important trade-offsControl and auditability versus speed of iterationFreshness versus indexing costIsolation versus utilization

When to keep it simple

If your workload is modest, a single well-run inference engine with prefix caching, a lakehouse or even a well-governed object store, and one hybrid search index will carry you a long way. Disaggregation is a response to measured interference and scale. A lakehouse is a response to governance, lineage and rebuild needs. Neither is a badge of maturity. My bias is to build the instrumentation and the data model first: tenant and identity everywhere, versioned chunks, latency budgets per stage, and cost tags. When the measurements show the limits, the next architectural step is usually obvious, and cheaper than guessing.

FAQ

What is disaggregated inference architecture?

It runs the two phases of LLM inference, prefill (processing the prompt) and decode (generating tokens), on separate, independently scaled GPU pools, with the KV cache moved between them. Each phase gets its own parallelism and capacity settings, and prefill work no longer interrupts token generation [1][5].

What is the difference between traditional and disaggregated inference?

In traditional (colocated) inference, prefill and decode share GPUs and often batches, so long prompts can delay streaming responses. In disaggregated inference they run on separate pools, which removes most of that interference and allows per-phase tuning, at the cost of a KV cache transfer step, a smarter router and more failure modes [1][5].

What are prefill and decode in LLM inference?

Prefill processes the entire prompt in one pass and builds the KV cache. It is compute-intensive and sets time to first token. Decode generates output one token at a time, rereading the weights and growing cache each step. It is memory-intensive and sets the pace of streaming [2].

Does prefill decode disaggregation improve throughput?

Not by itself. vLLM's documentation states that disaggregated prefill does not improve throughput. Its value is control: tuning time to first token and inter-token latency separately, and limiting tail inter-token latency when long prefills would otherwise interrupt decoding [5]. Any throughput or cost gain has to be measured on your workload.

When does disaggregated inference make sense?

It tends to help with long, variable prompts, strict targets for both time to first token and streaming smoothness, a fleet large enough for two right-sized pools, and a fast, measured interconnect. For short prompts, small traffic or an untuned colocated setup, prefix caching and chunked prefill are simpler first steps [5][8].

What is lakehouse RAG and when do enterprises need it?

Lakehouse RAG builds retrieval indexes from data held in open table formats such as Iceberg, Delta Lake or Hudi, so versions, lineage and access rules live in governed tables and indexes are rebuildable. Enterprises need it when they must answer where an answer came from, who may see it, and how to rebuild after a model change [19][21][22].

Why is a vector database alone not an enterprise knowledge architecture?

A vector database stores embeddings and answers similarity queries. It does not by itself hold the source of truth, record lineage, version documents, manage permissions across systems or let you rebuild indexes after an embedding upgrade. Those needs belong to a governed data layer, with the vector index as a derived, replaceable component.

How does retrieved context length affect GPU cost and KV cache?

Every retrieved token adds prefill compute, KV cache memory for the whole decode duration, and transfer bandwidth in a disaggregated setup. Because the KV cache per request grows with context, long contexts reduce how many requests a GPU can hold. Reranking to a smaller, better context is one lever to test first [4].

How do you prevent cross-tenant data leakage in RAG?

Apply permission filters inside the retrieval query so the index cannot return chunks the caller may not see, propagate identity through every stage, and use separate indexes or namespaces for high-isolation tenants. Add negative tests that must fail closed. OWASP lists sensitive information disclosure and vector and embedding weaknesses among LLM risks [25].

Which metrics should I track for LLM serving and RAG?

Track time to first token, time per output token, inter-token latency and end-to-end latency at p50, p95 and p99, plus tokens per second, GPU utilization, KV cache usage and prefix cache hit rate [6]. For RAG add retrieval latency, index freshness and answer quality. For money, track cost per successful query and cost per million generated tokens.

Where this leaves you

The two halves of an enterprise AI platform reward the same discipline. Treat data as governed and rebuildable, treat inference capacity as a measured, shared resource, and treat context length as the variable that connects them. The result is a platform whose costs are explainable, whose failures are bounded and whose security answers do not depend on prompt wording.

About the author. I am Asif Ali, a Principal AI & Enterprise Architect, Fractional CTO and AI Security & Governance specialist. My focus is production-grade AI architecture: the point where design decisions meet cost, security and operating reality.

If you are evaluating or modernizing AI infrastructure, a RAG platform, an inference stack or a GPU strategy, I enjoy comparing notes on the trade-offs with CTOs, founders and engineering leaders. Some of that thinking is in my approach to production-readiness assessments and AI security assessments, and you can book a conversation if a deeper exchange would be useful.

This article is educational. It presents a reference design and hypothetical scenarios, not benchmarks or client work. Product names are used only for identification and comparison, and no affiliation or endorsement is implied. Actual designs require assessment of your workloads, data, regulatory obligations and team. Implementing any control described does not by itself establish compliance.

References

  1. Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving, arXiv 2401.09670, January 2024.
  2. Patel et al., Splitwise: Efficient generative LLM inference using phase splitting, arXiv 2311.18677, November 2023.
  3. Qin et al., Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, arXiv 2407.00079, June 2024.
  4. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv 2309.06180, September 2023.
  5. vLLM documentation, Disaggregated Prefilling (experimental).
  6. vLLM documentation, Production Metrics.
  7. vLLM documentation, Benchmark CLI.
  8. vLLM documentation, Automatic Prefix Caching.
  9. NVIDIA, Dynamo documentation and the Dynamo project repository.
  10. NVIDIA, NIXL (NVIDIA Inference Xfer Library).
  11. SGLang documentation, PD Disaggregation (source of the documentation page).
  12. NVIDIA, TensorRT-LLM: Disaggregated Service.
  13. llm-d project, llm-d.
  14. Kubernetes SIGs, Gateway API Inference Extension.
  15. LMCache, LMCache documentation.
  16. Kubernetes documentation, Dynamic Resource Allocation.
  17. Kubernetes documentation, Multi-tenancy.
  18. Kubernetes documentation, Network Policies.
  19. Apache Iceberg documentation, Evolution.
  20. Apache Iceberg documentation, Branching and Tagging.
  21. Delta Lake documentation, Change data feed and Table batch reads and writes (time travel).
  22. Apache Hudi documentation, Timeline and SQL Queries (incremental and CDC queries).
  23. Elastic documentation, Reciprocal rank fusion.
  24. OpenSearch documentation, Hybrid search.
  25. OWASP GenAI Security Project, OWASP Top 10 for LLM Applications 2025.
  26. NIST, AI Risk Management Framework.
  27. NIST, AI RMF: Generative Artificial Intelligence Profile (NIST AI 600-1), 26 July 2024.
  28. European Commission, Regulatory framework for AI (the AI Act, Regulation (EU) 2024/1689).
  29. Kubernetes documentation, Schedule GPUs.
  30. NVIDIA, AIPerf.

Evaluating an AI inference or RAG platform?

If you are weighing inference architecture, RAG performance or GPU cost, I am glad to compare notes on the trade-offs or talk through a production-readiness review.