Quantization is “use fewer bits per weight”, but the interesting part is where it goes wrong.

The basic move

Map a float range to integers with a scale (and maybe a zero point):

q = round(x / scale)      # store q as int8
x ≈ q * scale             # dequantize on the fly

For a tensor whose values cluster nicely, int8 (256 levels) is plenty and you barely notice. The model gets ~4× smaller and, on hardware with int8 matmul, faster.

Outliers ruin the average

The failure mode in transformers is activation outliers — a few channels with values 10–100× larger than the rest. A single tensor-wide scale then has to stretch to cover the outliers, and every ordinary value collapses into a handful of quantization levels. Accuracy falls off a cliff.

Fixes, roughly in order of effort:

  • Per-channel scales instead of per-tensor — each channel gets its own range.
  • Keep outlier channels in higher precision (the LLM.int8() idea).
  • Smooth the outliers into the weights ahead of time (SmoothQuant).

fp8 vs int8

fp8 keeps an exponent, so it handles wide dynamic range more gracefully than int8 — which is exactly the outlier problem. That’s why newer accelerators lean on fp8 for inference. The rule of thumb I’ve landed on: weights tolerate int8 well; activations are where you want the float format.