What Quantization Costs You: the IQ1→Q8 Ladder With Measured Numbers

Home Rig Updated 2026-11-04 · 3 min read · #quantization #gguf #qat #quality

Status board (checked 2026-11-04)


What it is	weights stored in fewer bits, dequantized in small groups at compute time; the error shows up as KLD (distance from the original’s next-token distribution)
Flash-Next ladder (Vultr GGUF, KLD vs bf16 — lower is closer)	Q5_K_M 0.0325 · Q4_K_M 0.0361 · IQ3_M 0.0650 · IQ3_S 0.0665
Task-level	Qwen Coder IQ1_M (≈32 GB, keeps 256 of 512 experts): 91% of SWE-bench Verified, 99% of LiveCodeBench — vs the full model
GLM packs (Maya)	keep 97.7–99.2% of FP8 zero-shot accuracy; Maya-S picks the original’s next token 92.6% of the time (DeepSeek), 87.6% (MiniMax); Maya-L 90% (GLM)
QAT era	MiniMax M3.1 trains at NVFP4 (repo 252 GB vs 854 GB for M3); Kimi K3 uses MXFP4/MXFP8 from SFT onward; DeepSeek ships FP4 experts (278 of 510 GB) — the originals now come low-bit

What it is

Quantization trades storage for capacity: 4-bit-ish storage is what lets a 125B model fit in 37 GB or a 428B one in 93 GB — the entire home-rig trick stands on it. The bill comes in two currencies:

Distribution drift (KLD) — the model picks slightly different next tokens; visible in the ladder above as you drop from Q5 to IQ3;
Task degradation — uneven by design: chat survives low bits better than code and agents, where one wrong token derails a whole edit.

Getting started (3 steps)

Pick the largest quant that fits with 10–20% headroom — the RAM page gives the fit rule; a quant that swaps RAM pressure for SSD thrash loses more than it gains.
Prefer official low-bit when it exists. M3.1, Kimi K3 and DeepSeek are quantization-aware trained: their native 4-bit is closer to the original than any after-the-fact repack of the same size.
Score it on your task, not on vibes — run the same 10 real prompts on two quants and diff the outputs. Published tests (“matches the full model”) are maps, not the territory.

Best use

Situation	The measured answer
“Can I go below Q4 at all?”	Yes, with structure: the Coder build keeps 256/512 experts at IQ1_M and holds 91% SWE-bench — keeping experts beats keeping bits
“How bad is 3-bit really?”	KLD roughly doubles vs Q4_K_M (0.0361 → 0.0665) — fine for chat, risky for long agent runs
“Do I need to quant at all?”	8-bit/full packs exist for every model (Unsloth Q8 for DeepSeek is 508 GB) — that’s a 128 GB-class RAM conversation

Honest weaknesses

KLD is a proxy: it says “different”, not “worse” — but it correlates, and nobody publishes a free lunch.
Community repacks vary in quality and intent (the OrcaRouter “Uncensored” builds change refusal behavior and quant) — treat them as different models.
QAT narrows the ladder but doesn’t delete it: M3.1’s Maya-S still matches the original’s next token only 87.6% of the time.

FAQ

Does 4-bit make a model dumb?

The honest measured answer: barely, when it’s done with structure — 91% SWE-bench at 1-bit with expert-keeping, 97.7–99.2% zero-shot retention across GLM packs. The cliff is below 2 bits without tricks like expert-keeping or QAT.

Why do same-bit packs differ in size?

Hybrid and imatrix quants spend bits where importance scores say it matters (Vultr’s per-layer precision), and formats (GGUF vs EXL3 vs Maya packs) carry different overheads. Same bpw label, different file.

Where do I actually download these?

Each model page links its packs; the engines (Strata, Maya) ship their own formats — which is why the software choice comes before the download.

📌 Summarised from official docs and hands-on samples. Product features and pricing change often — official site wins.

← Newer: What GPU Do You Need to Run the Big Three at Home? VRAM Tiers, Checked November 2026 Older: Which Software Runs the Big Three? Strata, Maya, and the GGUF World, Compared →