Which quantized model actually fits in your card — and how fast will it run? Pick your GPU and a model to find out.
t/s ≈ bandwidth ÷ model size in VRAM. These are estimates, not guarantees — actual numbers shift with context length, batch size, and drivers.
| Quant | Bytes/param | 70B model ≈ | Quality vs FP16 |
|---|---|---|---|
| FP16 / BF16 | 2.00 | ~140 GB | Reference (lossless) |
| Q8_0 | 1.06 | ~74 GB | Near-lossless |
| Q6_K | 0.82 | ~57 GB | Excellent |
| Q5_K_M | 0.71 | ~50 GB | Very good |
| Q4_K_M | 0.61 | ~43 GB | Good — the sweet spot |
| Q3_K_M | 0.49 | ~34 GB | Noticeable degradation |
| Q2_K | 0.37 | ~26 GB | Significant loss — last resort |