How much GPU memory do you need to run an LLM?

By Zulu Calculator · Updated 19 October 2026 · AI & apps

To run a large language model, its weights, its KV cache and some working memory must all fit in GPU memory (VRAM). The weights take parameters × bits per weight ÷ 8 bytes, the KV cache grows with the length of the chat, and the software needs about 10–20% on top. For example, Llama 3.3 70B at Q4_K_M with an 8,192-token context needs about 46.15 GiB: two 24 GB RTX 4090 cards, or a 64 GB Mac.

This guide explains each part, how quantisation shrinks a model, why long chats and many users need more, how Macs and mixture-of-experts models differ, what fine-tuning adds, and how fast a model will write. To skip the maths, use the LLM VRAM calculator with your own model and GPU.

How do you calculate how much VRAM an LLM needs?

Add up three things. The weights are the model itself. The KV cache is the model's notes on every token of the conversation so far. Overhead is working memory for CUDA and the framework.

weights = parameters × bits per weight ÷ 8
KV cache = 2 × layers × KV heads × head size × bytes per number × context tokens × batch
overhead = (weights + KV cache) × overhead %
total = weights + KV cache + overhead
GPUs needed = total ÷ memory of one GPU, rounded up

GPU makers sell memory in GiB (1,073,741,824 bytes) but call it "GB". A "24 GB" card holds 24 GiB. Our figures are in GiB unless they say GB (one billion bytes).

What do FP16, Q8_0 and Q4_K_M mean?

They are ways of storing each weight with fewer bits. This is called quantisation. Fewer bits means less memory and faster writing, with some loss of quality at the low end. Q4_K_M is the usual pick for home use.

GGUF formats (used by llama.cpp, Ollama and LM Studio) keep embeddings and some layers at higher precision, so their whole-file average is a little above the name. These are the typical figures our calculator uses. All are approximate.

The figures above are weights only. Memory falls in step with the bits: Q4_K_M takes less than a third of FP16.

Memory for common model sizes

Weights in four formats, plus the full total at Q4_K_M with an 8,192-token context, FP16 KV cache and 10% overhead:

Why does a longer context need more memory?

For every token in the chat, each layer keeps a key and a value. That is the KV cache, and it grows in a straight line with the context length. Llama 3.1 8B has 32 layers, 8 KV heads and a head size of 128, so with an FP16 cache (2 bytes per number) it keeps 2 × 32 × 8 × 128 × 2 = 131,072 bytes per token. At 8,192 tokens that is exactly 1 GiB.

Some models keep the cache small. Gemma 3, gpt-oss and Llama 4 use sliding-window attention: most layers keep only the most recent tokens (in Gemma 3, five in six layers keep the last 1,024). DeepSeek V3 stores a compressed cache of 576 numbers per layer per token. That is why a 671B model needs less cache than an 8B one at long contexts.

Two settings shrink the cache further. An FP8 or Q8 cache halves it and a Q4 cache quarters it, with a small quality cost. And batch multiplies it: each chat running at the same time keeps its own cache. Four chats at 8,192 tokens on Llama 3.3 70B need 4 × 2.50 = 10.00 GiB of cache.

Worked examples

Example 1: Llama 3.3 70B at Q4_K_M, 8,192 tokens

FP16 KV cache, 10% overhead, one chat:

• Weights: 70.6 billion × 4.8 ÷ 8 = 42.36 billion bytes = 39.45 GiB
• KV cache: 2 × 80 layers × 8 KV heads × 128 × 2 bytes = 327,680 bytes per token; × 8,192 = 2,684,354,560 bytes = 2.50 GiB
• Overhead: 10% × (39.45 + 2.50) = 10% × 41.95 = 4.20 GiB
• Total: 41.95 + 4.20 = 46.15 GiB (49.55 GB)

One RTX 4090 has 24 GiB, so you need 46.15 ÷ 24 = 1.92, rounded up to 2 cards. A 64 GB Mac lets the GPU use about three quarters, 48 GiB, so it fits with about 96% in use. One 80 GB H100 fits it easily.

Example 2: Llama 3.1 8B at Q4_K_M on a 12 GB RTX 3060

• Weights: 8.03 billion × 4.8 ÷ 8 = 4.818 billion bytes = 4.49 GiB
• KV cache at 32,768 tokens: 131,072 × 32,768 = 4.00 GiB
• Overhead: 10% × (4.49 + 4.00) = 0.85 GiB
• Total: 9.34 GiB, which fits in 12 GiB

At the full 131,072 tokens the cache alone is 16.00 GiB and the total 22.54 GiB, which does not fit. The longest context that fits is what is left after weights and overhead, divided by the cache per token: 12 GiB ÷ 1.1 = 11.71 billion bytes, minus 4.818 billion of weights = 6.90 billion bytes; ÷ 131,072 = 52,608 tokens. With an FP8 cache (65,536 bytes per token) that doubles to about 105,200 tokens.

Example 3: a mixture-of-experts model on a Mac

Qwen3 30B-A3B has 30.5 billion parameters, but only 3.3 billion are used for each token. At Q4_K_M with 8,192 tokens:

• Weights (all of them): 30.5 billion × 4.8 ÷ 8 = 18.3 billion bytes = 17.04 GiB
• KV cache: 2 × 48 × 4 × 128 × 2 = 98,304 bytes per token; × 8,192 = 0.75 GiB
• Overhead: 10% × 17.79 = 1.78 GiB; total 19.57 GiB

A 24 GB Mac gives the GPU about two thirds, 16 GiB, so it does not fit. A 36 GB Mac (M4 Max) gives 24 GiB, so it does. Speed is where the model shines: each token reads only the active weights, 3.3 billion × 0.6 = 1.98 GB, plus 0.81 GB of cache, 2.79 GB in all. At 410 GB/s that is 410 ÷ 2.79 ≈ 147 tokens a second at most. The dense Qwen2.5 32B at the same settings (22.18 GiB) also fits on that Mac, but reads 21.65 GB per token, so it tops out at about 19 tokens a second.

Do mixture-of-experts models need less memory?

No. Models like DeepSeek V3, Mixtral, Qwen3 30B-A3B and gpt-oss need memory for all their weights, because any expert can be picked for the next token. What they save is reading: each token reads only the active weights, so they write much faster than their total size suggests. Size the memory on the total parameters and the speed on the active ones.

How much memory can a Mac use for an LLM?

Apple Silicon Macs share one pool of memory between the system and the GPU. By default macOS lets the GPU use about two thirds of it up to 36 GB and about three quarters above that. So a 16 GB Mac gives about 10.67 GiB, 24 GB gives 16, 36 GB gives 24, 64 GB gives 48 and 128 GB gives 96. A Mac is slower per GB than a big graphics card (a 64 GB M4 Max reads 546 GB/s against 1,008 GB/s for an RTX 4090), but it can hold models no single home card can.

How much VRAM do you need to fine-tune a model?

Training keeps much more than the weights. Our calculator uses these rules:

• Full fine-tuning (mixed precision, AdamW): 2 bytes of weights + 2 of gradients + 4 for an FP32 master copy + 8 for AdamW's saved state = about 16 bytes per weight, plus activations. 8-bit AdamW keeps 2 bytes of state instead of 8.
• LoRA: the model stays frozen at the precision you pick (16-bit in the table below), and small adapters on every attention and MLP layer are trained at 16 bytes per adapter weight.
• QLoRA: the same, but the frozen model is stored in 4-bit (about 4.25 bits per weight).
• Activations grow with sequence length and batch. Gradient checkpointing keeps only each layer's input, saving memory but training about 30% slower.

With gradient checkpointing on and 10% overhead. QLoRA is how a 70B model can be fine-tuned on one 48–80 GB card. For full fine-tuning, DeepSpeed ZeRO or FSDP split the load across GPUs.

How many tokens per second will I get?

When a model writes, it makes one token at a time and reads every weight it uses once for each. Reading is far slower than the maths, so memory bandwidth sets the top speed.

bytes read per token = active weights + KV cache for this chat
top speed (tokens a second) = memory bandwidth ÷ bytes read per token

In Example 1, each token reads 42.36 + 2.68 = 45.04 GB. An RTX 4090 reads 1,008 GB/s, so the limit is 1,008 ÷ 45.04 ≈ 22 tokens a second. With the model split by layers, the two cards take turns, so speed stays near one card's. A 64 GB Mac (M4 Max, 546 GB/s) tops out near 12, and an H100 (3,350 GB/s) near 74.

This is an upper limit: real software usually reaches about 50–80% of it. Serving several chats at once reads the weights once for all of them, so total throughput rises. On one H100, four 8,192-token chats of the same 70B model need 54.40 GiB and get at most about 63 tokens a second each, 252 in total. Check your own setup with the LLM tokens per second calculator.

What if the model doesn't fit?

1. Pick a smaller format. Going from Q8_0 to Q4_K_M cuts the weights by more than 40%.
2. Shorten the context or shrink the cache. An FP8 cache halves it.
3. Offload to system RAM. llama.cpp, Ollama and LM Studio can keep some layers in system RAM. It works, but system RAM is 10–30 times slower than VRAM, so speed drops sharply.
4. Add GPUs. Split by layers (cards take turns) or with tensor parallel in vLLM (faster, usually needs 2, 4 or 8 cards and a little extra memory on each).
5. Rent one. Compare hourly prices for H100, H200 and other cards with the GPU cloud cost calculator. Its prices were checked on 10 October 2026 and change often.

Common mistakes

• Counting only the weights. The KV cache and 10–20% overhead come on top. A 70B model at 128K tokens needs 40 GiB of cache alone.
• Using the nominal bits for GGUF. Q4_K_M averages about 4.8 bits per weight, not 4.
• Sizing a mixture-of-experts model on its active parameters. Memory follows the total; only speed follows the active part.
• Counting all of a Mac's memory. The GPU gets about two thirds or three quarters by default.
• Forgetting batch. Every chat running at once keeps its own cache.
• Mixing up GB and GiB. 46.15 GiB is 49.55 GB.

Quick summary

• Weights = parameters × bits per weight ÷ 8. Q4_K_M is about 4.8 bits; FP16 is 16.
• KV cache = 2 × layers × KV heads × head size × bytes × tokens × batch. It grows with context and users.
• Add 10–20% overhead, then divide by one GPU's memory and round up.
• MoE models need memory for all weights but read only the active ones.
• Macs give the GPU about ⅔ (up to 36 GB) or ¾ (above) of their memory.
• Full fine-tuning takes about 16 bytes per weight; QLoRA makes 70B possible on one big card.
• Top speed ≈ memory bandwidth ÷ bytes read per token.

Work out your own setup with the LLM VRAM calculator, see how fast it will write with the tokens per second calculator, or compare renting with the GPU cloud cost calculator.

Model shapes from each model's published configuration and GPU and Mac figures from makers' spec sheets, as used by our calculators (specs checked 10 October 2026). All figures are approximate estimates; real use depends on the software, its settings and the driver.

This guide explains the rules in plain English for planning. It is not legal, financial or flight-operations advice; check your contract, the official source linked above, your employer or your operator.