LLM Tokens per Second Calculator
See how fast an AI model will write on your hardware: the top speed its memory allows, a realistic estimate and how long a reply takes.
Sizes are approximate, from each model's published configuration.
For a mixture-of-experts model, the part used for each token. Same unit as above.
Typical whole-file averages, approximate. Q4_K_M is the usual pick for home use.
On your GPU's or Mac's spec sheet, often called "memory bandwidth".
Makers' spec-sheet figures, not checked here. Confirm yours before relying on it.
A rough assumption: share of the top speed real software reaches. Change it if you've measured yours.
1 token ≈ ¾ of an English word (rough).
Each new token also reads the cache for every earlier token, so long chats slow down.
1 KB = 1,024 bytes.
Prompt plus reply so far.
Each card reads its slice at the same time, so bandwidth adds up.
Rough assumption for time lost syncing cards. Only used with 2 or more.
Weights are read once for every chat in the batch.
Dense (not sparse) figure at your weights' precision, from the spec sheet. Leave blank to skip prompt speed.
Rough assumption for how much of that compute is used.
Tokens the model reads before it starts writing.
Same model, other memory speeds
Your model and settings at other bandwidths per GPU (with your number of GPUs), one chat at a time. Type your own list; tap a row to use it.
Separate with spaces or commas, up to 12.
Will it fit in memory? Check with the LLM VRAM calculator.
How your speed is worked out
Each new token reads the model's weights once, so memory bandwidth sets the top speed.
- 1
Bytes per token
8B × 0.6 B
= 4.80 GB - 2
Top speed
1,000 GB/s
÷ 4.80 GB
= 208 tok/s - 3
Real-world speed
208 × 60%
= 125 tok/s - 4
Your reply
500 ÷ 125
= 4.0 s
≈ 375 words
Read more: the formula, a worked example and what changes the answer
When a model writes, it makes one token (a word or part of a word) at a time, and for each one it reads every weight it uses from memory. Reading is far slower than the maths, so the memory bandwidth (how many gigabytes a second the chip can read) sets the top speed. Real software never quite reaches it, so we also show a realistic estimate at an efficiency you can change.
The formula
bytes per weight = bits per weight ÷ 8
bytes read per token = active parameters × bytes per weight + KV cache per token × tokens in the chat
top speed (tokens a second) = memory bandwidth × number of GPUs ÷ bytes read per token
realistic speed = top speed × efficiency (× multi-GPU scaling with 2 or more)
time for a reply = reply tokens ÷ realistic speed
words ≈ tokens × 0.75 (rough, English)
prompt speed = compute × GPUs × compute efficiency ÷ (2 × active parameters)
time to first token = prompt tokens ÷ prompt speed
batch speed = batch size ÷ (weights + batch size × KV cache) × bandwidth × efficiency, up to the prompt-speed limit
Worked example
An 8 billion parameter model at Q4_K_M (about 4.8 bits per weight) on a chip with 1,000 GB/s, at 60% efficiency, writing 500 tokens:
- Bytes per weight: 4.8 ÷ 8 = 0.6 bytes.
- Read for each token: 8,000,000,000 × 0.6 = 4,800,000,000 bytes = 4.80 GB.
- Top speed: 1,000 GB/s ÷ 4.80 GB = 208.33 tokens a second.
- Realistic: 208.33 × 60% = 125 tokens a second, or 1,000 ÷ 125 = 8 ms a token.
- A 500-token reply: 500 ÷ 125 = 4.0 seconds (500 ÷ 208.33 = 2.4 seconds at top speed), about 500 × 0.75 = 375 words.
- Words a minute: 125 × 0.75 × 60 = 5,625.
With the KV cache added (Advanced), using Llama 3.1 8B's shape (32 layers, 8 KV heads, head size 128, FP16 cache): 2 × 32 × 8 × 128 × 2 = 131,072 bytes a token; with 4,096 tokens in the chat that's 536,870,912 bytes, so each token reads 5.34 GB and the top speed falls to 1,000 ÷ 5.34 ≈ 187 tokens a second.
Prompt speed on a chip rated at 165 TFLOPS, at 50%: 165,000,000,000,000 × 0.5 ÷ (2 × 8,000,000,000) = 5,156.25 tokens a second, so a 1,000-token prompt takes 1,000 ÷ 5,156.25 = 0.19 seconds before the first token.
Things that change the answer
- Mixture-of-experts (MoE) models such as Mixtral, Qwen3 30B-A3B, DeepSeek V3 and gpt-oss read only their active weights for each token, so they write much faster than their total size suggests, but they still need memory for every weight.
- Long chats slow down: the KV cache is read for every earlier token. Models with grouped-query attention, sliding windows or compressed caches (DeepSeek) keep it small.
- Spilling to system RAM: if the model doesn't fit in GPU memory, the part in system RAM is read at that much slower speed. Check the fit with the VRAM calculator.
- Several GPUs: tensor parallel (vLLM, TensorRT-LLM) splits every layer, so bandwidths add up, minus syncing time. Splitting by layers instead (llama.cpp's default) makes the cards take turns, so speed stays near one card's: use 1 GPU here for that.
- Batches: serving many chats at once reads the weights once for all of them, so total throughput climbs with batch size until compute runs out, while each chat gets a little slower.
- Speculative decoding and other tricks can beat the simple limit by checking several guessed tokens in one read. They aren't modelled here.
Bytes per weight used (approximate)
| Precision | Bits a weight | Bytes a weight | 8B model |
|---|---|---|---|
| FP32 (full precision) | 32 | 4 | 32.00 GB |
| FP16 / BF16 (standard) | 16 | 2 | 16.00 GB |
| FP8 / INT8 | 8 | 1 | 8.00 GB |
| GGUF Q8_0 | 8.5 | 1.0625 | 8.50 GB |
| GGUF Q6_K | 6.6 | 0.825 | 6.60 GB |
| GGUF Q5_K_M | 5.7 | 0.7125 | 5.70 GB |
| GGUF Q4_K_M | 4.8 | 0.6 | 4.80 GB |
| 4-bit GPTQ / AWQ / NF4 / MXFP4 | 4.25 | 0.5313 | 4.25 GB |
| GGUF Q3_K_M | 3.9 | 0.4875 | 3.90 GB |
| GGUF Q2_K | 3 | 0.375 | 3.00 GB |
GGUF bits per weight are typical whole-file averages, shared with our VRAM calculator. GPU bandwidths in the picker are makers' spec-sheet figures, not checked here. Efficiency, multi-GPU scaling and compute efficiency are rough assumptions, not measurements: real speed depends on the software (llama.cpp, Ollama, LM Studio, vLLM, MLX), its settings, drivers and the prompt. Every result is an estimate.
Common questions
How do you calculate tokens per second for an LLM?
Each new token makes the model read all of its (active) weights from memory once, so the top speed is memory bandwidth ÷ bytes read per token. An 8B model at Q4_K_M (about 4.8 bits, 0.6 bytes a weight) is 4.8 GB, so 1,000 GB/s gives at most 1,000 ÷ 4.8 ≈ 208 tokens a second. Real software reaches only part of that; at a rough 60% it's 125 tokens a second.
Why is my real speed lower than the theoretical one?
The bandwidth on a spec sheet is a peak. Real runs lose time to reading the KV cache, software overheads, small matrix sizes and syncing between GPUs. Use the efficiency setting for this; if you have measured your own speed, divide it by the theoretical figure to find your efficiency.
Why do mixture-of-experts models run so fast?
A mixture-of-experts (MoE) model keeps all its weights in memory but uses only a few experts for each token, so it reads only the active weights. A 30B model with 3B active reads about a tenth of its size per token, so it writes about as fast as a 3B model, though it needs the memory of a 30B one.
How many words is 100 tokens?
For English text, a token is roughly ¾ of a word, so 100 tokens is about 75 words. This is a rough rule: code, numbers and other languages use more tokens per word.
What is prefill and time to first token?
Before writing, the model reads your whole prompt in one go (prefill). That part is limited by compute, not memory: about 2 × active parameters floating-point operations per prompt token. Time to first token is the prompt length ÷ prefill speed. A long prompt on a slow chip can take many seconds before the first word appears.
Related calculators
Made by Zulu Calculator, part of AlphaZulu LLC. Formulas are shown in full and rule figures are checked against official sources; see how we build calculators.