Quantization notes: int8, fp8 and what you actually lose
Where the bits go, why outliers ruin everything, and the per-channel trick that saves them.
Inference
KV cache, and why LLM inference is memory-bound
The cache that makes autoregressive decoding fast also makes it the thing that runs out of memory first.