LLM Training Compute Calculator
Work out the FLOPs, GPU-hours, days and cost to train or fine-tune a language model from its size and the tokens it learns from.
Total tokens the model sees. 1 token ≈ ¾ of an English word (rough).
20 tokens per parameter is the Chinchilla paper's rule of thumb (Hoffmann et al. 2022).
With more than 1, the tokens above are your dataset size.
100% for full fine-tuning. LoRA trains a small share, often well under 5% (rough).
Your GPU's dense (not sparse) figure at your training precision, often BF16, from its spec sheet.
A rough assumption: share of peak compute the model really uses. Change it if you've measured yours.
Splits the GPU-hours into wall-clock time.
Your quote or rental price. Compare clouds with the GPU cloud cost calculator.
Your figure, including its share of the server if you know it. Leave blank to skip energy.
Building power ÷ IT power, for cooling and the rest. 1.2 is a rough assumption.
Only if you pay for power yourself.
Same settings, other model sizes
Your GPU, utilisation, GPU count and price for models from 1B to 70B parameters.
| Model | Tokens | FLOPs | GPU-hours | Time | Cost |
|---|
Will the model fit for training? Check the memory with the LLM VRAM calculator.
How your training compute is worked out
FLOPs per token × tokens, ÷ what your GPUs really deliver, then spread across them and priced.
- 1
FLOPs per token
6 × 7B
= 4.2 × 10¹⁰ - 2
Total compute
× 1T tokens
= 4.2 × 10²² FLOPs - 3
GPU-hours
÷ (312 TFLOPS
× 40%) ÷ 3,600
= 93,483 GPU-h - 4
Time and cost
÷ 64 GPUs = 61 days
× $2.00 a GPU-h
= $186,966
Read more: the formula, a worked example and notes
Training a language model means showing it tokens (pieces of words) and, for each one, running the model forward, then running backwards to work out how every weight should change. Each step costs floating-point operations (FLOPs). Add them up, divide by how many FLOPs your GPUs really get through each second, and you have the time; multiply GPU-hours by the hourly price and you have the cost.
The formula
FLOPs per token ≈ 6 × parameters (the widely used approximation for dense transformers: 2 forward + 4 backward)
fine-tuning with only some weights learning ≈ 4 × parameters + 2 × trainable parameters (approximate)
tokens = tokens typed × epochs (or 20 × parameters, compute-optimal)
total FLOPs = FLOPs per token × tokens
useful FLOPs a second per GPU = peak TFLOPS × 10¹² × utilisation (MFU)
GPU-hours = total FLOPs ÷ useful FLOPs a second ÷ 3,600
wall-clock hours = GPU-hours ÷ number of GPUs
cost = GPU-hours × price per GPU-hour
energy (kWh) = watts per GPU × GPU-hours ÷ 1,000 × PUE
Worked example
A 7 billion parameter model trained from scratch on 1 trillion tokens, on GPUs rated at 312 TFLOPS, at 40% utilisation, on 64 GPUs at $2 a GPU-hour:
- FLOPs per token: 6 × 7,000,000,000 = 42,000,000,000 (4.2 × 10¹⁰).
- Total compute: 4.2 × 10¹⁰ × 1,000,000,000,000 = 4.2 × 10²² FLOPs, which is 4.2 × 10²² ÷ (10¹⁵ × 86,400) = 486.11 petaflop/s-days.
- One GPU really does 312 × 10¹² × 0.40 = 1.248 × 10¹⁴ FLOPs a second.
- GPU-hours: 4.2 × 10²² ÷ 1.248 × 10¹⁴ = 336,538,461.5 seconds ÷ 3,600 = 93,482.9 GPU-hours (93,483 rounded).
- On 64 GPUs: 93,482.9 ÷ 64 = 1,460.67 hours ÷ 24 = 60.86 days (about 61).
- Cost: 93,482.9 × $2 = $186,965.81.
- Speed: 1.248 × 10¹⁴ × 64 ÷ 4.2 × 10¹⁰ = 190,171 tokens a second across all 64 GPUs.
- Tokens per parameter: 1,000,000,000,000 ÷ 7,000,000,000 = 142.86, against the Chinchilla rule of thumb of 20 (140 billion tokens for 7B).
Energy (Advanced), at 700 W a GPU and a PUE of 1.2: 700 × 93,482.9 ÷ 1,000 = 65,438.03 kWh × 1.2 = 78,525.64 kWh (about 78.5 MWh). At 0.10 a kWh that's 7,852.56 in electricity.
LoRA fine-tuning (Advanced) of the same 7B model on 100 million tokens with 1% of weights learning: (4 × 7,000,000,000 + 2 × 70,000,000) × 100,000,000 = 2.814 × 10¹⁸ FLOPs, which is 2.814 × 10¹⁸ ÷ 1.248 × 10¹⁴ ÷ 3,600 = 6.26 GPU-hours. Full fine-tuning on the same tokens is 4.2 × 10¹⁸ FLOPs (9.35 GPU-hours).
Things that change the answer
- Utilisation (MFU) matters most and is the hardest to know in advance. Communication between GPUs, small batches, data loading and recomputing activations to save memory all lower it. It's a rough assumption here: measure a short run and use your own figure.
- Long context: 6 × parameters leaves out attention's own maths, which grows with sequence length. At very long contexts the true count is higher.
- Mixture-of-experts models do the maths only for their active parameters, so use the active count here, not the total.
- LoRA: the backward pass still has to carry gradients through every frozen layer to reach the adapters, so only the weight-gradient part (2 per parameter) shrinks. That's why we use 4 × parameters + 2 × trainable, not a tiny fraction of 6. This is an approximation.
- Failures and restarts, evaluation runs and experiments before the final run aren't counted. Real projects usually spend more than one clean run.
6 × parameters × tokens is the widely used approximation for dense transformers (Kaplan et al. 2020, "Scaling Laws for Neural Language Models"). 20 tokens per parameter is the Chinchilla paper's rule of thumb (Hoffmann et al. 2022, "Training Compute-Optimal Large Language Models"). GPU compute, utilisation, power and PUE are your own figures or rough assumptions, not measurements. Every result is an estimate.
Common questions
How many FLOPs does it take to train an LLM?
A widely used approximation for dense transformers is 6 × parameters × training tokens: about 2 FLOPs per parameter per token for the forward pass and 4 for the backward pass. A 7 billion parameter model trained on 1 trillion tokens needs about 6 × 7×10⁹ × 10¹² = 4.2 × 10²² FLOPs.
How do you turn FLOPs into GPU-hours?
Divide the total FLOPs by what one GPU really delivers each second: its peak TFLOPS × 10¹² × your utilisation (MFU), then divide by 3,600. At 312 TFLOPS and 40%, one GPU does 1.248 × 10¹⁴ useful FLOPs a second, so 4.2 × 10²² FLOPs takes about 93,483 GPU-hours.
What is MFU?
Model FLOPs utilisation: the share of a GPU’s peak compute that goes into the model’s own maths. Time spent waiting on memory, communication between GPUs, data loading and recomputing activations all lower it. The 40% default here is a rough assumption; measure your own run and change it.
How many tokens should I train on?
The Chinchilla paper (Hoffmann et al. 2022) found that, for a fixed compute budget, about 20 tokens per parameter gave the best model. Many recent models train on far more than that, because a smaller model trained longer is cheaper to run afterwards.
How much cheaper is LoRA fine-tuning?
Less than you might think in compute. The forward pass and the backward pass through the activations still run over the whole model; only the weight-gradient part shrinks to the small adapters. So FLOPs fall from about 6 to about 4 per parameter per token. The big savings with LoRA are in memory and in needing far fewer tokens.
Related calculators
Made by Zulu Calculator, part of AlphaZulu LLC. Formulas are shown in full and rule figures are checked against official sources; see how we build calculators.