KV cache: plan GPU memory for context and concurrency
KV cache makes a language model's context part of the GPU memory budget. A model can load successfully and still leave too little room for the requests a service needs to handle. Longer prompts, longer answers, and more simultaneous generations all change that calculation. We approach capacity planning by separating model fit from serving capacity, then translating the workload into a cache budget.
What is KV cache?
A key-value (KV) cache stores attention keys and values from tokens a transformer model has already processed. Attention is the mechanism that lets a token use information from earlier tokens. Keys help determine which earlier information is relevant; values carry the information attention combines.
During prefill, the model processes the prompt and builds its initial cache. During decode, it generates a continuation, reusing cached keys and values and appending new ones as tokens are processed. Each attention layer maintains its own state. This avoids recomputing earlier attention state at every generation step.
Full attention still reads the retained history when computing attention for a new token. Caching removes repeated work; the cost of using a longer history remains.
Model weights and KV cache have different lifetimes
Weights are the model's learned parameters. Requests using the same loaded model can share those weights. Their attention state normally differs because their prompts and continuations differ.
| Memory component | What it holds | What changes its size |
|---|---|---|
| Model weights | Learned parameters shared by requests in a replica | Model architecture, weight precision, and placement across GPUs |
| KV cache | Attention state for retained tokens | Context length, active sequences, KV-head count, and cache precision |
| Runtime memory | Intermediate calculations, execution buffers, and bookkeeping | Serving software, execution settings, and workload shapes |
A replica is one serving instance of the model, on one GPU or a cooperating group of GPUs. Adding another replica requires another weight allocation, while admitting more requests to an existing replica mainly adds request state and execution work.
For our fully managed clusters, we treat context and concurrency as architecture requirements alongside GPU, network, and storage requirements. The serving runtime, the software executing the model, determines how KV state is allocated within that infrastructure.
Why context and concurrency multiply memory demand
For a model using full attention, the cache grows with the tokens retained by each active sequence. Count the complete formatted input, including system instructions, conversation history, retrieved documents, and tool results. Then allow room for the output still to come.
Concurrency means sequences whose state must remain resident, rather than registered users or open client connections. One request can also generate several candidate continuations, increasing its state requirements. Continuous batching, which adds and removes sequences as generations progress, changes which requests are active without removing their need for cache capacity.
The useful planning unit is resident tokens across the replica. Ten sequences holding 4,000 tokens each and two holding 20,000 each have the same raw KV payload when the model and cache precision are identical. Their latency and compute requirements can differ.
Architecture matters, too. Grouped-query attention (GQA) lets several query heads share key/value heads. A head is one attention subspace processing a representation of the tokens. Storage follows the number of KV heads, which can be much smaller than the query-head count.
Calculate KV-cache size from the model architecture
For a standard decoder with identical full-attention layers and equal key/value head dimensions:
KV bytes per retained token = 2 × layers × KV heads × head dimension × bytes per element
The leading 2 counts keys and values. Head dimension is the number of elements in each head's key or value vector. BF16, or bfloat16, and FP16, or 16-bit floating point, each use two bytes per element.
Multiply that per-token cost by the sum of retained tokens across sequences:
KV payload = bytes per token × total retained tokens
This is tensor payload, before allocation overhead or duplicate storage. Paged allocation, which stores state in fixed-size blocks, rounds each sequence's allocation to block boundaries. Shared prefixes can reduce duplication when the runtime shares their physical blocks. These are the allocation and sharing mechanisms behind PagedAttention's block management.
Worked example: an 8,192-token request needs 1 GiB
Consider Mistral-7B-Instruct-v0.3. Its published model configuration specifies 32 layers, 8 KV heads, 32 query heads, and a hidden size of 4,096. That gives a head dimension of 4,096 ÷ 32 = 128. This configuration has no sliding attention window and specifies a maximum sequence length of 32,768.
Our illustrative deployment assumptions are:
- One NVIDIA L40S GPU with 48 GB of advertised device memory.
- One model replica, with BF16 weights and a BF16 cache.
- A 16 GiB KV budget, after separate allowances for weights, runtime memory, and operational headroom. This budget is an assumption, not a measurement of free memory on the card.
- One continuation per request, with no prefix sharing, cache quantization, or offload.
The calculation is independent of a particular inference-engine version. It estimates storage, rather than measured throughput or a supported deployment configuration. KiB and GiB are binary units: 1 KiB is 1,024 bytes, and 1 GiB is 1,073,741,824 bytes.
2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token
A 6,144-token formatted prompt with a 2,048-token output allowance therefore needs:
8,192 × 128 KiB = 1 GiB of planned KV payload
The same model and cache precision produce these memory-only ceilings:
| Planned tokens per sequence, input plus output | KV payload per sequence | Payload for 8 sequences | Sequences in a 16 GiB KV budget |
|---|---|---|---|
| 4,096 | 0.5 GiB | 4 GiB | 32 |
| 8,192 | 1 GiB | 8 GiB | 16 |
| 16,384 | 2 GiB | 16 GiB | 8 |
| 32,768 | 4 GiB | 32 GiB | 4 |
These ceilings consume the full assumed budget and exclude metadata and other allocator costs. They are upper bounds, not admission settings or latency guarantees. Increasing the planned sequence length from 8,192 to 32,768 cuts the payload-only concurrency ceiling from 16 to four, without changing the weights.
Mixed lengths need an admission budget
For identical planned lengths, the memory-only ceiling is the KV budget divided by each sequence's planned payload, rounded down. Mixed traffic needs a sum of the individual requirements.
In the same example, two sequences planned for 32,768 tokens consume 8 GiB. Eight more planned for 8,192 tokens consume another 8 GiB. Ten sequences now exhaust the payload budget, even though a short-request-only calculation suggested 16 would fit.
Future output belongs in admission control, the decision to start, queue, or reject work. Dynamic allocation may initially admit many short prompts. If their answers grow together, they can later compete for the same remaining blocks. A workload contract therefore needs both a concurrency limit and a token-growth allowance.
Allocated GPU memory is different from active cache pressure
A serving process can reserve a large KV pool before requests fill it. High process-level GPU memory use alone does not show how much active generation state occupies that pool.
Separate three categories:
- Active blocks (green) hold state that running sequences still need.
- Reusable, unreferenced blocks (gray) preserve earlier prefixes for possible future requests. They can be evicted without interrupting a running sequence, although a later request may lose a cache hit.
- Free blocks (outlined) are immediately available for new state.

The distinction matters operationally: evicting an idle prefix costs a potential future saving; preempting active work pauses progress and may require state transfer or recomputation.
A cache hit also has a scope. Reuse requires compatible model state and a matching token prefix, with any tenant or security boundaries enforced by the implementation. Similar wording alone does not make attention state interchangeable.
Prefix retention should have its own budget. Reuse patterns and effective eviction policies depend on workload, as observed in production serving traces. Keeping more historical prefixes is not automatically worth displacing active capacity.
Choose the optimization that addresses the limit
Start with the resource being exhausted. Weight memory, active KV state, reusable prefixes, and execution time call for different changes.
| Observed constraint | Practical response | Tradeoff |
|---|---|---|
| Active state approaches the KV budget as answers grow | Limit combined input/output tokens and concurrent sequences | Some requests wait or need smaller inputs |
| Short requests occupy oversized allocations | Use dynamic or paged allocation where supported | Block rounding and allocation overhead remain |
| Many requests repeat the same leading tokens | Reuse compatible prefixes within the permitted sharing scope | Idle prefixes compete for space and can be evicted |
| Weights leave little room for request state | Reduce weight precision or change model placement | Weight quantization does not reduce KV bytes per token |
| Active KV payload is the limiting allocation | Reduce cache precision where supported | Numerical behavior and execution overhead can change |
| Retained state exceeds GPU capacity | Move eligible state to CPU memory or storage where supported | Transfers add latency and consume host or storage bandwidth |
| Memory fits, but response latency is unacceptable | Reduce active work or add serving capacity | More memory alone does not remove execution bottlenecks |
Cache precision is separate from weight precision. Reducing a two-byte cache element to one byte halves the raw tensor payload; scales, metadata, and implementation details can reduce the net saving. There is no corresponding promise of twice the throughput.
Offloading changes where state lives and how it reaches the GPU. Moving idle prefixes out of GPU memory and repeatedly transferring active layer state have different latency consequences. The Transformers cache strategies describe offloading's memory/speed tradeoff and warn that quantized caches can hurt latency at short contexts. Neither is a universal first fix.
Where the simple formula stops applying
Sliding-window attention retains a bounded recent history in the layers using it. Hybrid models can mix those layers with full attention, so their cache has both bounded and growing components. Other architectures need their own storage accounting; the model's architecture determines the formula.
Static caches can reserve a configured maximum even when current sequences are short. Dynamic caches grow with the retained state. Across multiple GPUs, the budget must fit the actual per-device placement; adding all fleet memory together does not create a shared cache pool.
Put the capacity envelope into the cluster specification
We define a useful serving target with a model and revision, weight and cache precision, input/output limits, active-sequence limits, and a latency objective. The specification also needs the serving-software version, per-device memory allowances, prefix-retention policy, and the owner of each layer. A memory-only ceiling is the starting bound; sustainable throughput is the rate that still meets the latency objective.
The cluster layer has a different responsibility from the cache allocator. clusterdOS is the open-source engine inside each cluster. It is built on Kubernetes, and can ship enterprise-grade Slurm and VMs. Kubernetes coordinates containerized workloads, Slurm schedules batch compute jobs, and virtual machines (VMs) provide guest operating-system environments.
Our fully managed clusters cover architecture, deployment, and end-to-end operations. Those responsibilities do not make a theoretical KV calculation a validated configuration. Failure recovery, facility loss, and provider commitments belong to the separate infrastructure reliability plan.
Talk to an engineer about a workload-specific cluster built around your model, context, concurrency, and latency requirements.