How many GPUs to run Llama 3 70B?
Llama 3 70B needs about 140GB of GPU memory at FP16 precision. That means two 80GB GPUs, such as 2x A100 80GB or 2x H100 80GB. At INT8 precision it needs about 70GB and fits on one A100 80GB. At INT4 it needs about 35GB.
Updated 2026-10-28
GPU memory by precision
The memory a model needs is set by its parameter count and the number of bits used per parameter. Llama 3 70B has 70 billion parameters. The table applies that arithmetic to the three precisions used for inference.
| Precision | Bytes per parameter | GPU memory for 70B |
|---|---|---|
| FP16 / BF16 | 2 | about 140GB |
| INT8 | 1 | about 70GB |
| INT4 | 0.5 | about 35GB |
These are weight-only figures. Add room for the KV cache, activations and framework overhead. The real footprint is higher than the table. Keep a safety margin when you size the hardware.
GPU choices for each precision
| Precision | GPU count | Example hardware |
|---|---|---|
| FP16 | 2 GPUs | 2x A100 80GB or 2x H100 80GB |
| INT8 | 1 GPU | 1x A100 80GB or 1x H100 80GB |
| INT4 | 1 GPU | 1x A100 80GB, or 2x 48GB cards for more headroom |
Two smaller cards can replace one large card when the total memory is enough. Four 48GB cards give 192GB, which covers FP16 weights with room for the KV cache. The trade-off is more interconnect traffic and more failure points.
For a full cost comparison of GPU rental against API access, use the self-host vs API break-even calculator. For a per-token view of the same models, see the cheapest way to run Llama page.
Related questions
Can I run Llama 3 70B on a single consumer GPU?
A 24GB consumer GPU cannot hold the FP16 or INT8 weights. It cannot hold the INT4 weights of about 35GB either. A single 48GB workstation card can hold the INT4 weights.
Does quantization change output quality?
INT8 usually keeps quality close to FP16. INT4 loses more quality and is best for experiments and low-stakes workloads.