cat Quantizing LLM Weights and the KV Cache.md

Two Memory Pools, Not One

It is easy to think of quantization as one job: make the model smaller. That view is incomplete. Modern LLM inference has two separate memory pools, and quantization targets them independently:

1. Model weights: the static parameters. The footprint is fixed and does not change with load.
2. The KV cache: the key and value tensors that attention stores for every past token. This footprint is dynamic and grows with batch × context length.

These are not competing options. They target two different budgets, and real serving stacks run both. So the question is never “which one” but “how much of each.”

Here is the 2025–2026 state for both, and how they combine.

Why Quantization Pays Off at All

Quantization reduces the numeric precision of tensors from a wide format (usually FP32 / FP16 / BF16) to a narrower one: INT8, INT4, FP8, FP4, or sub-2-bit. The trade is universal: less memory, more arithmetic throughput, and lower memory-bandwidth pressure, at the cost of some accuracy loss and quantization/dequantization overhead.

The key fact is that modern LLM inference is overwhelmingly memory-bandwidth bound, not compute bound. Each decode step must read the model weights and the entire KV cache from VRAM. Cutting bytes-per-value therefore translates directly into tokens/second and tokens per dollar, often more than raw FLOPs do.

There are two families of technique:

	PTQ (Post-Training)	QAT (Quantization-Aware)
When	After training, on a finished model	During training or fine-tuning
How	Convert weights with a small calibration set (or none)	Simulate quantization in the forward and backward pass (“fake quantization”)
Effort	Minutes to hours	Days, with data and compute
Accuracy at low bits	Good to ~4-bit; degrades below	Far better at ≤4-bit and ≤2-bit
Use	Almost all production serving	Shipping ultra-low-bit models (BitNet, gpt-oss FP4, DeepSeek-V4 experts)

A third distinction dominates the 2025–2026 story: numeric format.

• Integer (INT8, INT4): classic, general-purpose, works on nearly all hardware.
• Low-precision floating-point (FP8, FP4): hardware-native on modern GPUs (Hopper H100/H200 has FP8; Blackwell B200 has FP8 and FP4). FP8 keeps better accuracy than INT at the same bit-width because the exponent preserves dynamic range and outliers. This single property is why FP8 won.

Weight Quantization

Weight quantization compresses the static model parameters. The model size in VRAM is roughly params × bytes-per-weight. A few concrete numbers:

Model	FP16	INT8	INT4 / FP4
7B	14 GB	7 GB	~4 GB
70B	140 GB	70 GB	~35 GB

Four-bit puts a 70B model on a single 48 GB consumer card. That is the breakthrough that made “big models at home” possible.

The mature, production methods

Method	Bits	Idea	Quality	Best for
FP8 (W8A8)	8 (float)	Weight + activation to FP8; native on Hopper and Ada	~99%	Production GPU serving on Hopper+; the 2026 default
INT8 / W8A8	8 (int)	Weight + activation; SmoothQuant smooths outliers	97–99%	Broad hardware compatibility (no FP8)
AWQ	4	Activation-aware: protect the salient top ~1% of weights	~95%	Creative writing, coding, latency-sensitive serving
GPTQ	4	Second-order PTQ; huge pre-quantized model library	~90%	Max throughput, especially with Marlin kernels (2.5× faster)
GGUF (llama.cpp)	2–8	Single-file format; importance-weighted K-quants (Q4_K_M etc.)	~92%	CPU / Apple Silicon / edge
bitsandbytes NF4	4	On-the-fly quant; the QLoRA fine-tuning standard	Good	Fine-tuning (QLoRA), quick loads

Four method-level insights worth remembering:

• AWQ’s core idea: not all weights matter equally. It scales and protects the ~1% of salient weights (identified from activations), so 4-bit loses almost nothing.
• GPTQ + Marlin: the kernel matters more than the algorithm. Marlin runs the same GPTQ weights 2.5× faster with optimized CUDA kernels.
• GGUF K-quants allocate more precision to the important parts of each tensor block. Q4_K_M (≈ 4.5 bits/weight) is the popular sweet spot; Q5_K_M for better quality.
• SmoothQuant solved activation outliers, which is what made W8A8 (weight and activation INT8) work reliably.

What 2025–2026 added

1. FP8 became the production standard. FP8 is native on NVIDIA Hopper (H100/H200) via the Transformer Engine and on AMD MI300X/MI355X via ROCm. vLLM and SGLang serve FP8 with no calibration required (dynamic BF16→FP8). Typical numbers: a 0.1–0.3% perplexity increase, ~33% faster inference, ~50% less weight and activation memory.

2. FP4 arrived on Blackwell (B200): two competing 4-bit floating formats.

• NVFP4 (NVIDIA): a 4-bit mantissa group of 16 elements plus an FP8 microscaling exponent. DeepSeek-R1-0528 quantized to NVFP4 stays within ~1% of FP8 across MMLU-Pro, GPQA, and LiveCodeBench, at ~2.3× higher throughput. DeepSeek-V4 (a 1.6 T-parameter MoE) trains its expert weights directly in FP4 (QAT). gpt-oss ships in MXFP4.
• MXFP4 (OCP standard): a block of 32 plus a shared FP8 exponent. It runs on AMD MI355X (ROCm MFMA) and Blackwell. “Enhanced MXFP4” closes to within ~1% of NVFP4 fidelity with ~6% GEMM overhead.

The hardware caveat matters: FP4 needs Blackwell. On Hopper or Ampere, AWQ-INT4 or GPTQ-INT4 remain the 4-bit options.

3. Sub-2-bit went from research to “almost there.” Rotation-based PTQ removes outliers without changing the model:

• QuaRot (randomized Hadamard rotations) and SpinQuant (ICLR 2025, learned rotations). SpinQuant beats QuaRot, LLM-QAT, and SmoothQuant, narrowing the gap to full precision on the hard-to-quantize Llama-3 8B.
• QuIP#: incoherence preprocessing plus vector quantization → the first viable 2-bit-per-weight results.
• AQLM: additive vector quantization, sub-2-bit.

4. Ternary (1.58-bit) and sub-1-bit. BitNet b1.58 (weights ∈ {-1, 0, 1}, log₂3 ≈ 1.58 bits) needs QAT from scratch. 2026 work made ternary post-training: CAT-Q (2026) produces ternary models from just ~512 calibration samples that beat BitNet-v1/v2 trained on 100 B tokens. NanoQuant is the first sub-1-bit PTQ method.

5. FP8 weight + activation is now fused into serving engines. vLLM’s quantization=fp8 loads pre-quantized FP8 weights and runs FP8 GEMMs end-to-end.

Which weight quant to pick

Scenario	Recommended	Why
Production GPU, Hopper (H100/H200)	FP8	Near-lossless (~99%), ~33% faster, native HW
Production GPU, Blackwell (B200)	FP8, or NVFP4/MXFP4 for max density	FP4 ~2.3× throughput at ~1% drop
Production GPU, Ampere (A100)	AWQ or INT8	No native FP8
Max raw throughput	GPTQ + Marlin kernels	Best tokens/s on NVIDIA
Quality-critical (creative, code)	AWQ or FP8	Best accuracy retention
Local / laptop / CPU / Apple Silicon	GGUF Q4_K_M / Q5_K_M (llama.cpp, Ollama)	Hardware-democratic; K-quants balance quality
Mobile / edge	GGUF Q4	Memory efficiency
Fine-tuning on small GPUs	bitsandbytes NF4 (QLoRA)	Standard 4-bit LoRA recipe
Research, extreme compression	QuaRot / SpinQuant / QuIP# (2-bit)	Push density below 2 bits
Shipping a new ultra-low-bit model	QAT (BitNet-style ternary or FP4 QAT)	PTQ breaks at these bits

KV Cache Quantization

Why the KV cache dominates at long context

The KV cache stores the K and V tensors of every past token in every attention layer, so autoregressive generation avoids recomputing attention from scratch each step. Memory per token is approximately 2 (K+V) × 2 bytes × num_layers × num_kv_heads × head_dim.

For Llama-2-7B, 10 k tokens of context is about 5 GB of KV cache, nearly a third of the weights. It gets worse fast:

• A 70B model at 8k context uses about 20 GB of KV cache per request; a batch of 32 is about 640 GB.
• The KV cache routinely exceeds the model weights in long-context serving. At 128k+ tokens it is the single largest VRAM consumer.
• Each decode step must read the entire cache, so decoding is memory-bandwidth bound and inter-token latency (ITL) grows linearly with context length.

So KV cache quantization has two compounding payoffs:

1. Capacity: more concurrent requests, or longer supported context, on the same GPU.
2. Speed: halving bytes per cached token halves the memory traffic per attention step, directly lowering the ITL slope.

The foundational insight: K and V want different granularity

The most-cited result in KV cache quantization, from KIVI (2024) and reinforced by Atom, is a granularity asymmetry, not a bit-width one. Keys and values are best quantized along different axes:

• Quantize K per-channel (group along the channel dimension). Keys have a few fixed channel outliers, the same activation-outlier effect SmoothQuant found, so each outlier channel needs its own scale.
• Quantize V per-token (group along the token dimension). The reason is attention sparsity: attention scores are highly sparse, so per-token grouping keeps each token’s quantization noise confined to itself instead of bleeding into the few high-attention tokens.

A common point of confusion: KIVI uses the same 2 bits for K and V. What fails at extreme low bit is same granularity (both per-token), not same precision. At FP8 this distinction barely matters. FP8’s dynamic range handles both K and V near-losslessly, which is exactly why vLLM’s FP8 KV cache (identical precision for K and V) is the production default and works essentially perfectly. The per-channel / per-token split only becomes essential when you push the cache to 4-bit and below.

Method	Bits	Idea	Note
KIVI (2024)	2	Tuning-free asymmetric 2-bit; K per-channel, V per-token	Enabled 8× larger batch, ~2× throughput; foundational
KVQuant (NeurIPS 2024)	2	2-bit, 1.5× smaller than KIVI at equal accuracy; pre-fetch kernels	Targets 10M-token context
Atom (2024)	4	Mixed-precision 4-bit KV; flags key outliers	High-throughput
HF Transformers (quanto/HQQ)	2/4/8	int2/int4/int8 + residual cache (keep recent ~128 tokens in FP)	int4 ≈ fp16 quality; int2 degrades

What 2025–2026 added

1. FP8 KV cache went production-default. The vLLM blog (Apr 2026) stress-tested --kv-cache-dtype fp8 (e4m3) across Hopper and Blackwell and concluded it is “ready to be the default starting point for many long-context deployments.” Validated results:

• Halves KV cache memory; decode ITL slope drops to ~54% of BF16 (Llama-3.1-8B on H100), with break-even at ~7k tokens.
• Near-lossless: 97–99% accuracy recovery on reasoning (AIME25, GPQA, MATH500) and long-context benchmarks (mrcr up to 128k–1M), even uncalibrated (scale=1.0).
• Under load (concurrency 8, ~20k input): +14.9% output throughput, lower median ITL.
• Bugs found and fixed: Flash-Attention-3 lost precision at long context (FP8 accumulation error → a 128k needle-in-haystack test crashed from 91% → 13%). The fix was two-level accumulation (back to 89%). The team also added per-head scales, layer skipping for hybrid models, and fused query quantization.
• On Blackwell (B200) with FlashInfer the accumulation bug is gone; FP8 KV still wins.

2. FP4 KV cache on Blackwell. For a given long-context workload, the memory shrinks nicely: BF16 ≈ 43 GB → FP8 ≈ 21.5 GB → NVFP4 ≈ 10.7 GB. FP4 KV needs Blackwell.

3. Low-bit KV (TurboQuant, 2026). vLLM shipped turboquant_k8v4 (FP8 keys + 4-bit values, 2.6× compression, +1.17% PPL), turboquant_4bit_nc, and 3-bit variants (3.8× compression but +2.7% PPL). The vLLM study’s verdict: FP8 remains the recommended default. TurboQuant 4-bit and 3-bit trade accuracy and actually slow down due to dequant overhead, except in KV-bound regimes. AMD’s Triton/FlyDSL TurboQuant kernels add ~3.6× speedup.

4. Mixed-precision and sensitivity-aware KV. Rather than one bit-width for all layers:

• KVTuner (ICML 2025): multi-objective search for per-layer K/V precision pairs; near-lossless at lower average bits.
• MixKVQ (ACL 2026): query-aware mixed-precision (BF16/INT4/INT2 per request).
• QAQ, MiKV, ZipCache: dynamically identify critical KV entries and keep them higher-precision on the fly.

5. Sparsification as an alternative or complement. RocketKV (NVlabs, ICML 2025) does not quantize; it simply does not store the KV entries that attention will not look at (selective sparsity).

6. The 2026 frontier moved beyond scalar quant. Latent-space compaction (Attention Matching, ~50×) and reasoning-aware compression (TriAttention, ~10.7× memory reduction on AIME25 at matched accuracy) point at where the field is heading.

7. Hybrid-attention handling matured. Models with small sliding-window layers (gpt-oss-20b) should keep those layers in BF16; their KV is bounded, so quantization adds overhead with no savings. vLLM’s --kv-cache-dtype-skip-layers sliding_window is now the recommended recipe for hybrid models.

When to use FP8 KV cache (and when not to)

Use it when the workload is decode-heavy, memory-bound, or long-context, the common production case. Avoid it (keep BF16) when:

• Contexts are short (< ~7k tokens): the small constant FP8 overhead is not amortized.
• head_dim = 256 and prefill latency matters: two-level accumulation raises TTFT ~1.6×.
• Uncalibrated accuracy drops below ~95% on your workload (some FlashMLA models): calibrate instead, or stay BF16.
• The model has many small sliding-window layers: skip those layers, keep the rest FP8.

How the Two Complement Each Other

Weight quantization and KV cache quantization target two different memory budgets. Here is the direct comparison.

Dimension	Weight quantization	KV cache quantization
What it compresses	Static model parameters	Dynamic per-token attention state (K, V of past tokens)
Memory character	Fixed, same regardless of load	Variable, grows with batch × context_length
Primary benefit	Fit a bigger/cheaper model per GPU; faster compute (more low-prec FLOPs)	More concurrent users and/or longer context; faster decode (less memory traffic)
When it dominates	Short context, few users (weights are most of VRAM)	Long context, high concurrency (cache exceeds weights)
Cost model	One-time (quantize once, serve forever)	Continuous (every token quantized and dequantized live)
Best 2026 default	FP8 (Hopper+); AWQ/GPTQ elsewhere; GGUF on CPU	FP8 (Hopper/Blackwell)
Risk if naive	Accuracy loss at <4-bit	Accuracy loss + dequant overhead; wrong granularity below 4-bit

The combination wins, with one narrow caveat

Two memory pools, additive wins. FP8 weights are ~2× smaller weights; FP8 KV is ~2× smaller cache. Stacked, they roughly halve both the fixed and the variable VRAM cost. You can fit a bigger model and serve more users or longer context than either alone. A 70B model at 32k context drops from ~640 GB-class cache pressure to tens of GB once both pools are FP8 or FP4.

Match the quant to the bottleneck. If your pain is “the model will not fit,” lead with weight quant. If your pain is “I cannot fit enough concurrent users or long-enough context,” lead with KV quant. Most production serving feels both, so run both.

One narrow pitfall to know. There is a genuinely fragile case: aggressive low-bit KV quantization with the wrong granularity (both K and V per-token) on top of already weight-quantized models. One study saw perplexity blow up to thousands there. The fix is the KIVI granularity rule (K per-channel, V per-token); the bit-widths can still be equal. This is an edge case at the research frontier, not something you hit with production FP8. The more practical combination cost is different: naive, non-fused setups can slow generation ~3× from dequant overhead, which is why you use an engine with fused kernels.

Use an engine that fuses both. This is why vLLM, SGLang, and llama.cpp beat hand-rolled stacks. Their FP8 weight + FP8 KV path is a single fused pipeline (FP8 GEMM → FP8 attention with two-level accumulation, per-head scales, layer skipping). You get both halvings without the overhead penalty. The 2026 production recipe is essentially:

vllm serve <model> \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  # ... plus PagedAttention, prefix caching,
  #     continuous batching, and chunked prefill

Hardware gates both. FP8 (weights and KV) wants Hopper+; FP4 wants Blackwell. On Ampere or CPU you fall back to AWQ/GPTQ-INT4 (weights) and INT4/INT8 KV, the same principle with integer flavors.

The 2026 State at a Glance

	Weights	KV cache
Compresses	Static params (fixed)	Dynamic per-token K/V (scales with batch × ctx)
Production default (2026)	FP8 (Hopper+); AWQ/GPTQ-INT4 elsewhere	FP8 (Hopper/Blackwell)
Extreme density	NVFP4 / MXFP4 (Blackwell)	NVFP4 KV (Blackwell); TurboQuant 4-bit
CPU / edge	GGUF K-quants (Q4_K_M / Q5_K_M)	INT4/INT8 KV (llama.cpp)
Research frontier	Sub-2-bit (QuaRot/SpinQuant/QuIP#), ternary BitNet/CAT-Q/NanoQuant	Mixed-precision (KVTuner, MixKVQ), sparsify (RocketKV), latent compaction
Headline 2026 win	DeepSeek-V4 experts trained in FP4; gpt-oss ships MXFP4	vLLM FP8 KV = 2× cache, ~54% ITL slope, 97–99% accuracy to 1M ctx
Combinability	Multiplies with KV quant	Multiplies with weight quant; use fused kernels, and K per-channel / V per-token granularity below 4-bit

Summary

The whole field fits in one sentence:

Quantize weights to fit the model; quantize the KV cache to serve the load. Run both through an engine with fused kernels. FP8 (identical precision for K and V) is the safe default; reserve K per-channel / V per-token granularity for pushing the cache below 4-bit.

Weight quantization gives you a bigger or cheaper model per GPU. KV cache quantization gives you more concurrent users or longer context on the same GPU. FP8 is the 2026 default for both on Hopper and Blackwell, and it uses identical precision for K and V with near-lossless results. FP4 (NVFP4/MXFP4) is the emerging extreme-density option on Blackwell. AWQ, GPTQ + Marlin, and GGUF K-quants remain the right tools on older hardware and at the edge. The one nuance to remember: when you push the KV cache below 4-bit, switch to K per-channel and V per-token, a granularity change rather than a precision change, and let an engine with fused kernels handle it.

If you run local LLMs through vLLM or SGLang, the practical takeaway is simple: turn on quantization=fp8 and kv-cache-dtype=fp8 together, and let the fused pipeline do the work.