KV Cache Size Calculator

Compute the exact KV cache memory a transformer model holds from the numbers in its official config.json. The calculator uses bytes per token = 2 × layers × KV heads × head_dim × bytes per element. It also has a reverse mode (how much context fits a memory budget) and an MHA vs GQA comparison. Numbers update live as you type.

KV cache size
—
Enter your inputs above.
—
Bytes per token
—
Per sequence
—

Formula: bytes per token = 2 × layers × KV heads × head_dim × bytes per element. The leading 2 counts the K and V tensors. GiB uses 1,073,741,824 bytes; GB uses 1,000,000,000 bytes.

Where the formula comes from

During self-attention, every layer stores a key tensor and a value tensor for each token in the context. With grouped-query attention, several query heads share one KV head, so the cache holds num_key_value_heads × head_dim numbers per tensor per layer per token. In FP16 each number is 2 bytes, and there are two tensors, so bytes per token = 2 × num_hidden_layers × num_key_value_heads × head_dim × bytes per element. Worked check from the official Qwen3-8B config (36 layers, 8 KV heads, head_dim 128, BF16): 2 × 36 × 8 × 128 × 2 = 147,456 bytes per token, which is 4.5 GiB at 32,768 tokens.

Sliding-window models can cap the cache

Two of the presets, Mistral-7B-v0.1 and Phi-3-mini-4k-instruct, declare an active sliding window in their config, which can cap the cache. The Qwen2.5 configs list sliding_window 131072, but they set use_sliding_window to false. The Hugging Face transformers documentation says: "Note that for models using sliding window attention (Mistral, Gemma2,…) or chunked attention (Llama4), the cache will stop growing when the layers using these types of attention have reached their maximum size (the sliding window or chunk size)." Mistral-7B-v0.1 declares "sliding_window": 4096 and Phi-3-mini-4k-instruct declares "sliding_window": 2047. At context lengths above the window, the cache of the sliding-window layers can stop growing at the window size, so the result above can be an upper bound for these two models.

What the result does not include

The result is the KV cache only. Model weights, activations, CUDA context and framework overhead are extra. How much extra depends on the runtime and settings, so this page does not apply a fixed overhead percentage.

Why GQA matters so much

Qwen2.5-72B-Instruct declares 64 attention heads but only 8 KV heads. If the same model used classic multi-head attention, it would store 64 KV heads per layer instead of 8, eight times the cache. Phi-3-mini-4k-instruct uses full multi-head attention (32 KV heads at head_dim 96), so its 393,216 bytes per token exceed the 327,680 bytes per token of Qwen2.5-72B-Instruct.

Config values used in the presets

Copied from the official Hugging Face config.json of each model (fetched 2026-11-17). Where a config omits head_dim, it is hidden_size ÷ num_attention_heads. Sources: Qwen3-8B, Qwen3-32B, Qwen2.5-7B-Instruct, Qwen2.5-72B-Instruct, Mistral-7B-v0.1, Phi-3-mini-4k-instruct.

ModelLayersKV headsAttention headshead_dim
Qwen3-8B36832128
Qwen3-32B64864128
Qwen2.5-7B-Instruct28428128
Qwen2.5-72B-Instruct80864128
Mistral-7B-v0.132832128
Phi-3-mini-4k-instruct32323296

KickLLM Margin Studio · Offline analysis app

Know your AI costs. Now plan your margin.

Turn your usage CSV into a cost breakdown, test revenue and growth assumptions, and export a report for your next pricing decision. Margin Studio runs locally with rates you supply.

Get Margin Studio, $39 once Try the interactive preview → One-time purchase · Downloadable ZIP