kvcached GPU Memory Sharing: How Elastic KV Caching Works

The Static Allocation Bottleneck in Production LLM Serving

Modern large language model (LLM) inference engines such as vLLM and SGLang rely on PagedAttention or RadixCache to handle token generation. However, both engines suffer from an operational constraint rooted in GPU memory allocation: static reservation. By default, these engines claim a fixed, dominant share of physical GPU memory at initialization (typically 80% to 90% via flags like --gpu-memory-utilization) to allocate their KV cache pools upfront.

This design prevents memory overcommit, but it imposes severe architectural penalties on multi-tenant infrastructure:

1. Restricted allocation flexibility: Fixed cache reservations can constrain how capacity is divided between colocated models.
2. Stranded Memory During Inactivity: An idle model retains its entire KV cache buffer in physical VRAM, preventing other services or background training tasks from utilizing available memory.
3. Overprovisioning for Peak Load: Infrastructure teams must dimension GPU clusters for worst-case concurrency, leaving significant high-bandwidth memory (HBM) unutilized during intermittent traffic troughs.

kvcached addresses this bottleneck by importing classic operating system virtual memory concepts directly into LLM runtime architectures. By decoupling GPU virtual memory address ranges from physical page allocations, it makes physical KV allocation demand-driven for supported integrations. Shared demand still has to fit the physical device.

Core Architecture: Decoupling Virtual Addressing from Physical Storage

Traditional LLM serving treats GPU memory allocations as contiguous physical commitments managed via standard CUDA runtime allocators or PyTorch caching allocators. kvcached introduces a virtual memory daemon and C++ extension layer (vmm_ops) that interfaces directly with low-level CUDA Virtual Memory Management (VMM) APIs.

Instead of requesting physical memory blocks immediately, engines augmented with kvcached allocate an expansive virtual address space at startup. Physical memory pages (by default 2 MiB) are committed only when an inference request actually requires a KV cache block for attention computation. Once generation completes or a model transitions to sleep mode, kvcached unmaps and releases the physical pages back into the shared pool without altering the virtual address layout.

Solving Kernel Crashes: The Zero-Page Mechanism

A critical technical obstacle in virtualizing GPU KV caches is the mismatch between memory page boundaries. The PagedAttention block size used inside inference engines does not necessarily match the 2 MiB physical memory page granularity recognized by GPU memory management hardware. Furthermore, optimized attention kernels and static CUDA graphs often perform speculative reads across unallocated memory offsets.

In standard CUDA execution, dereferencing an unmapped virtual address triggers an immediate memory access violation and crashes the process. To solve this, kvcached introduces a zero page mechanism. Any reserved virtual address not currently backed by an active, physical cache block is mapped to a shared, read-only, zero-filled physical page. When attention kernels speculatively read unallocated slots, they receive harmless zeros instead of illegal address faults. The documented mechanism addresses speculative reads into unused slots. It does not establish crash-free CUDA graph replay for every engine, kernel or model combination.

Integration and Operational Setup

Integrating kvcached into existing production environments uses import interception for supported integrations; pin and validate the engine version rather than assuming that the patch is maintenance-free.

Prerequisites

• OS / Platform: Linux with CUDA or ROCm (AMD HIP) environments.
• Python: 3.10 through 3.13.
• PyTorch: >= 2.10 (built against the PyTorch stable ABI).
• Engines: SGLang (tested up to v0.5.20) or vLLM (tested up to v0.30.0).

Launch Configuration

Because kvcached manages physical allocation dynamically, the documented launch path changes how KV capacity is allocated. Keep total physical demand and application admission limits explicit; elastic KV allocation does not eliminate other memory budgets.

Advanced Features and Hardware Nuances

1. Prefix Caching Compatibility: Traditional memory ballooning or virtualization techniques frequently conflict with prefix reuse. As documented in the project repository, kvcached maintains compatibility with Automatic Prefix Caching (APC) in vLLM and RadixCache in SGLang. Unused prefixes can be bounded or evicted gracefully in supported integrations, with behavior to validate against the selected engine revision.

2. Dynamic Page Sizing for Hybrid Models: While standard attention configurations use a 2 MiB page size (KVCACHED_PAGE_SIZE_MB), newer hybrid architectures (such as Qwen3.5, Qwen3.8, or models using Mamba state slots) may have single blocks or state slots exceeding 2 MiB. If KVCACHED_PAGE_SIZE_MB remains unset, kvcached automatically scans the block geometry and selects the smallest safe multiple of 2 MiB. In SGLang, attention pools and Mamba state pools retain distinct page sizes, preventing state slot padding from artificially expanding standard attention pages.

3. AMD ROCm Backend Specifics: Hardware backends process KV memory layouts differently. On NVIDIA CUDA hardware, kvcached defaults to a contiguous KV layout. However, on AMD ROCm (HIP) builds, kvcached automatically defaults to a non-contiguous KV-cache layout (KVCACHED_CONTIGUOUS_LAYOUT=false). This prevents errors where vLLM's ROCm attention kernels receive strided per-layer KV tensors that they cannot decode correctly.

Performance and Deployment Realities

In multi-tenant benchmarks serving three Llama-3.1-8B models concurrently on a single NVIDIA A100-80GB GPU under intermittent burst workloads, kvcached achieved between 2x and 28x Time to First Token (TTFT) reduction compared to standard static reservation engines. The reported comparison concerns that particular multi-model burst setup; it does not establish a universal delay or an infeasibility claim for every static serving configuration.

Furthermore, Red Hat's AI engineering group integrated kvcached into Sardeenz, an enterprise controller designed for dynamic multi-model serving inside Kubernetes and OpenShift environments.

Architectural Tradeoffs and Constraints

• Upstream Engine Drift: Because kvcached relies on monkey-patching internal engine memory structures (KVCACHED_AUTOPATCH=1), breaking changes in rapid release cycles of vLLM or SGLang can disrupt hook points. Upgrades to underlying engines must be validated against tested compatibility tags (e.g., SGLang v0.5.20, vLLM v0.30.0).
• Virtual Address Space Saturation: Record allocation and mapping behavior for the chosen number of colocated processes; the reviewed sources do not establish a universal process-count or virtual-address limit.
• Allocation Jitter Under Severe Contention: When aggregate multi-model concurrency genuinely exceeds physical VRAM limits, demand-driven allocation must handle eviction or throttling. Teams deploying kvcached in latency-critical SLAs must pair it with front-end routers and admission control to prevent concurrent Out-Of-Memory (OOM) aborts.

Decision Framework: When to Deploy kvcached

For engineering teams operating homogeneous, dedicated clusters running a single saturated foundational model, standard static reservation remains the simplest path. But for compound AI systems, dynamic Kubernetes multi-tenancy, and bursty endpoints, kvcached provides the virtual memory layer necessary to reclaim underutilized GPU capacity.

Evaluate elastic allocation under both quiet and overlapping load

Consider three model services whose bursts normally occur at different times. kvcached's demand-driven KV allocation is interesting because inactive cache reservations may strand capacity, but shared physical VRAM remains finite. Elastic cache architecture

Keep model weights and other runtime allocations in the budget; freeing KV pages does not free every allocation in the process. Compare the same workload against the static configuration and distinguish first-token delay from throughput. Pin the engine and patch integration to a tested combination before evaluating upgrades.

A virtualization mechanism can improve allocation flexibility without making simultaneous demand fit when it exceeds physical capacity. This is a proposed capacity experiment, not a reproduction of the reported TTFT range or a guarantee of OOM-free execution.

Evidence and scope

Primary repository documentation was refreshed on 14 October 2026. The worked scenario is a planning exercise derived from documented behavior, not a product test. Reported measurements remain attributed to their source and workload.

• kvcached README
• ovg-project/kvcached repository metadata
• ovg-project/kvcached | DeepWiki
• Solve the GPU Cost Crisis with kvcached