LLM GPU Memory Requirements: Weights, KV Cache and OOM

A model that loads and answers a short prompt has passed one capacity check. It has not demonstrated that several long requests can finish together. For autoregressive inference, stored keys and values let later tokens reuse earlier attention work; their storage is separate from the model's weights. Transformers caching explanation

Start with a budget for each GPU, not the sum of memory across a machine:

W + K_allocated + R_peak + H ≤ C

• W — resident weights: the tensors actually placed on this device, including any quantization metadata.
• K_allocated — cache allocation: physical storage committed to the cache, including unused capacity within reserved buffers or blocks.
• R_peak — other runtime memory: budget for activations, temporary workspaces, graph pools and other allocations outside W and K. Establish it for the configurations and request shapes you allow.
• H — headroom: an explicit planning allowance for observed variation and operational requirements, not an assumed universal percentage.
• C — capacity available to this process: after accounting for other device users. Do not substitute a product's advertised capacity without checking its units and actual availability.

This is a planning decomposition, not an engine's accounting API. Avoid counting the same storage twice when reconciling it with allocator metrics. PyTorch documents memory retained by its allocator and private pools used by CUDA graphs; vLLM also documents graph capture as an additional memory cost. PyTorch 2.9 CUDA semantics, vLLM 0.11.0 memory guidance

Scope: text-generation inference with a conventional decoder attention cache. Training, multimodal encoders, speculative draft models and unusual state representations need additional accounting. Evidence was checked on 10 October 2026. Implementation examples below are pinned to Transformers 4.57.1, vLLM 0.11.0, PyTorch 2.9 and Accelerate 1.10.1; these are reference versions, not claims about the latest release.

For a full-attention decoder with the same cache dimensions in every layer, start with the live-token payload implied by the stored tensors:

KV_payload_bytes = 2 × L × H_kv × D × b_kv × Σ T_i

The tensor dimensions are documented in Transformers caching. Use KV heads, not automatically the number of query heads: the pinned Llama attention implementation projects keys and values using num_key_value_heads and updates the cache before the attention operation. Grouped-query attention can therefore store fewer heads than it uses for queries.

Count the rendered prompt, including templates and retrieved text, plus the generation allowance. For admission planning, prompt length plus the maximum permitted output is a useful conservative token bound; the precise current cache length depends on the generation step. Count resident sequences, not registered users or requests still waiting outside the engine. Multiple completions or beams need an implementation-specific expansion of that count.

This formula is a payload estimate, not total allocated memory. If an implementation materializes a dense batch cache, replace Σ T_i with that tensor's batch and allocated sequence dimensions. The vLLM paged-attention layout stores tokens in blocks, so partially used blocks still occupy storage. Padding, metadata and separately retained reusable prefixes can add capacity requirements. Assume no prefix sharing in a conservative first estimate.

Change the formula when its premises change. Sliding-window or chunked layers have bounded retained history in the documented Transformers caches; mixed layer types need a layer-by-layer sum. Static caches reserve capacity ahead of use. Other state representations need their own documented layout. Transformers cache strategies

Hypothetical arithmetic, not a GPU test or a claim about a named model. Assume one device, all weights resident, exactly 8,000,000,000 parameters at 2 bytes each, 32 full-attention layers, 8 KV heads per layer, 128 elements per head, and a 2-byte KV cache. Assume four independent sequences, no prefix sharing, and no additional draft model.

Each sequence is allowed 12,288 prompt tokens and 4,096 output tokens: a planning bound of 16,384 cached tokens. Here, 1 KiB = 1,024 bytes and 1 GiB = 1,073,741,824 bytes; decimal GB is a different unit.

That remainder must accommodate cache-allocation overhead, the rest of the runtime and headroom. No evidence here establishes that 1.10 GiB is enough. A successful weight load therefore cannot approve this four-request configuration.

Now change one premise at a time:

These are changes to estimated demand. An engine with a preallocated pool may retain the same physical allocation after demand falls. Reconcile the estimate with its configured pool and available cache blocks.

Once you have a defensible runtime allowance and headroom, an initial full-attention admission bound is:

total resident token budget ≤ floor((C − W − R_peak − H) / KV_bytes_per_token)

Use bytes throughout. This bound assumes ideal packing; subtract allocation overhead and other retained cache state, then respect the engine's actual cache capacity. It is a candidate to validate, not a safe concurrency promise.

Use the budget to select a change, then check its application cost. These controls can be combined; they are not competing product categories.

The weight/cache distinction is explicit in the documentation: TGI quantization describes weight formats and loading, while vLLM's FP8 cache documentation describes a separate KV representation and its scaling factors. Do not infer a four-bit cache from four-bit weights.

For the pinned Transformers version, offloaded caches move layer state between host and device, and quantized caches can add latency. Cache strategies Accelerate separately documents weight offload and warns about host-memory and disk-transfer constraints. Accelerate 1.10.1 The graph option is documented in vLLM 0.11.0.

More GPUs can change placement too, but recompute the budget per device from the chosen parallelism layout. A total that fits across several cards does not establish that each card's weights, cache and working memory fit. vLLM parallelism guidance

The following is a diagnostic procedure derived from the budget, not a guarantee that a symptom has one cause. Keep model revision, precision and request shape fixed while investigating one change.

In vLLM 0.11.0 V1, cache pressure can cause preemption and later recomputation. Its tuning guide describes a preallocated cache controlled through a GPU-memory fraction. Raising that fraction reallocates a finite budget; it does not create memory. The same guide distinguishes the scheduler's token-per-iteration budget from per-sequence length and sequence-count controls. Do not substitute max_num_batched_tokens for a bound on all tokens retained across requests. vLLM tuning

For PyTorch 2.9, inspect both allocated tensor memory and allocator-reserved memory; they overlap and must not be added together. Device-wide usage can include allocations outside those tensor counters. empty_cache() releases unused allocator-held memory, not live weights or KV tensors. A persistent excess of live demand therefore needs a budget change. CUDA memory management

If the suspect component remains unresolved, record that gap. An OOM message alone is not evidence for a universal overhead percentage or a recommended GPU size.

Before selecting hardware, fill in the facts that determine the budget:

1. Pin the model and runtime. Record model revision, attention/state architecture, layer count, stored KV heads and dimensions, weight representation, cache dtype, engine version and device placement. Verify supported combinations against that version's documentation.
2. Define admitted work. Record maximum rendered input and output tokens, simultaneous resident sequences, any beams or multiple completions, and the queue/rejection policy. Keep maximum context support separate from the workload you intend to serve.
3. Build a per-device worksheet. Separate W, cache payload, allocated cache capacity, other runtime memory and headroom. Use bytes consistently. Record where each number comes from: configuration, arithmetic or measurement.
4. Plan representative validation. Include startup, warm-up, longest admitted prompts, long generation and overlapping requests at the intended concurrency. Exercise the boundary case together; separate single-request and short-prompt checks do not cover their combination.
5. Capture the limiting evidence. Measure allocation peaks and engine cache capacity/occupancy; retain errors, preemptions and queue behavior. Include host RAM and transfer behavior if offloading. Use these observations to set runtime allowance and headroom.
6. Approve an operating envelope. Record the model/runtime configuration, request limits and device arrangement that passed your checks. Check answer quality and latency after precision, context or offload changes. Revalidate when any budget premise changes.

No hardware validation was performed for this guide. Its worked numbers identify what to measure and which changes affect the estimate. The deployment decision comes from checking the complete admitted workload on the intended stack.