How much VRAM does a local LLM need?

Last updated 2026-10-14 · by Prajyot Mulik

Try the tool: Local LLM VRAM Calculator

“Will this model run on my GPU?” has a more precise answer than “it depends”. Memory use during inference is dominated by two things you can calculate from numbers published in the model’s configuration: the weights and the KV cache. This guide gives the formula, then works it through for five open-weight models. You can follow every step in the Local LLM VRAM Calculator.

The formula

total memory = weights + KV cache + runtime overhead

Weights = number of parameters × bits per weight ÷ 8 bytes. A 7.6-billion-parameter model stored at 16 bits per weight needs about 15.2 billion bytes, which is 14.19 GiB. Quantisation lowers the bits per weight and therefore the size.

KV cache = 2 × layers × KV heads × head dimension × bytes per element × context tokens × simultaneous sequences. During generation the model keeps a key and a value vector for every token it has seen, in every layer; the leading 2 is that key-and-value pair. This part grows linearly with context length, which is why a long context can need more memory than the quantised weights.

Overhead covers the runtime’s own buffers, such as the GPU context, activation workspace and memory fragmentation. It is not a constant. The calculator defaults to 1 GiB as an assumption you can change, and you should measure your own runtime.

All sizes here are in GiB (1,073,741,824 bytes), which is the unit GPU memory figures normally use.

Where the inputs come from

The architecture numbers come from each model’s config.json on Hugging Face: num_hidden_layers, num_key_value_heads, hidden_size and num_attention_heads. The head dimension is hidden_size ÷ num_attention_heads for these models, because their configs have no separate head-dimension field. Parameter counts come from the safetensors totals Hugging Face publishes for each repository.

Quantisation: what the labels mean

Quantised GGUF files store weights in blocks with a scale, so the real size is a little above the nominal bit width. The llama.cpp quantisation README lists measured bits per weight on Llama-3.1-8B: F16 16.0005, Q8_0 8.5008, Q6_K 6.5633, Q5_K_M 5.7036, Q4_K_M 4.8944 and Q3_K_M 3.9960. Other models quantise to slightly different sizes, so treat these as good approximations rather than exact promises. Lower precision saves memory but can reduce output quality; this tool measures size, not quality.

Grouped-query attention changes the KV cache

Attention layers have a number of query heads, but many modern models store keys and values for far fewer “KV heads” and share them across query heads (grouped-query attention). The KV cache is proportional to KV heads, not attention heads. Compare two of the models below at 8,192 tokens of context with an fp16 cache:

• Qwen2.5-7B: 28 layers, 4 KV heads, head dimension 128 → 2 × 28 × 4 × 128 × 2 bytes = 57,344 bytes per token → 0.4375 GiB.
• Phi-3.5-mini: 32 layers, 32 KV heads, head dimension 96 → 2 × 32 × 32 × 96 × 2 = 393,216 bytes per token → 3.0 GiB.

Phi-3.5-mini has about half the parameters of Qwen2.5-7B, yet its KV cache is almost seven times larger because it has no grouped-query sharing. That is why you cannot rank models by parameter count alone.

Worked examples

Weights (GiB) at each quantisation, and KV cache at 8,192 tokens (fp16), from the calculator’s data:

Adding the 1 GiB overhead allowance, the totals at Q4 with an 8,192-token context are roughly 5.8 GiB for Qwen2.5-7B, 6.1 GiB for Mistral-7B, 6.2 GiB for Phi-3.5-mini, 10.9 GiB for Qwen2.5-14B and 21.7 GiB for Qwen2.5-32B. So the 7B and mini models fit an 8 GiB card with a short context, the 14B model suits a 12 or 16 GiB card, and the 32B model needs a 24 GiB card with little to spare.

Now stretch the context. Qwen2.5-7B at 32,768 tokens has a KV cache of 1.75 GiB, which still fits easily. Phi-3.5-mini at 32,768 tokens needs 12.0 GiB of KV cache alone, more than five times its Q4 weights. Context length matters.

Things the formula does not cover

• Runtime differences. llama.cpp, vLLM, Ollama and MLX allocate memory differently and some keep extra buffers or pre-allocate for the maximum context.
• Quantised KV caches. Some runtimes can store the cache in 8 bits. The calculator offers a one-byte approximation that ignores scale overhead.
• Batching. Serving several users at once multiplies the KV cache by the number of simultaneous sequences.
• Other model types. Mixture-of-experts models, multimodal encoders and LoRA adapters need extra terms; the five listed models are dense text models.
• Maximum context. Each model is configured for a maximum context (32,768 for the Qwen2.5 and Mistral models above, 131,072 for Phi-3.5-mini). The calculator warns when you go beyond it.

A short checklist

1. Find parameters, layers, KV heads and hidden size in the model’s config.json.
2. Pick a quantisation and compute weights.
3. Decide the context you really need; compute the KV cache.
4. Add 1 to 2 GiB for overhead and compare with your card.
5. If it does not fit, lower the context, choose a smaller quantisation, or offload layers to system memory and accept slower generation.

Sources

All pages were read on 14 October 2026.

• llama.cpp: quantize README – bits per weight by quantisation type
• Hugging Face config.json files: Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, Qwen2.5-32B-Instruct, Mistral-7B-Instruct-v0.3, Phi-3.5-mini-instruct
• Hugging Face model API – parameter counts (safetensors totals), shown for one model; the others follow the same URL pattern