LLM quantization explained: what Q4_K_M, Q8_0 and the rest mean

Quantization stores a model’s weights with fewer bits, so the model takes less memory and runs faster. Open models are usually released at 16 bits per weight. Q8_0 cuts that to about 8.5 bits and is close to lossless, while Q4_K_M uses about 4.9 bits, a little under a third of the original size, for a small but measurable drop in quality. That trade is why Q4_K_M is the default in most local tools.

The practical rule is to run the largest model that fits your memory at Q4_K_M or better, with room left for context. If you have memory to spare, step up to Q5_K_M, Q6_K or Q8_0. Drop below 4 bits only when it lets you run a much larger model. The rest of this guide explains the names, shows what published measurements say about quality at each level, and covers the newer 4-bit floating point formats (MXFP4 and NVFP4) and Apple’s MLX.

Why smaller also means faster

When a model generates a token, it reads every active weight from memory once. On a GPU that read is the bottleneck, so generation speed is roughly memory bandwidth divided by the size of the weights. Halve the bytes and you come close to doubling the tokens per second. llama.cpp’s own reference numbers for Llama 3.1 8B show the effect: generation ran at about 29 tok/s at 16-bit, 51 tok/s at Q8_0 and 72 tok/s at Q4_K_M on the same machine, according to its quantization README.

Prompt processing, where the model reads your input, depends more on raw compute and behaves differently. For chat, generation speed is the part you feel.

How to read GGUF quant names

GGUF is the model file format used by llama.cpp, Ollama and LM Studio. The quant name in a GGUF file tells you how the weights were packed:

• Q plus a number means quantized, at roughly that many bits per weight. Q4 is about 4 bits, Q8 about 8.
• _0 and _1 are llama.cpp’s original formats. Weights are split into small blocks that share a scale (and for _1, an offset). Q8_0 is the one still in wide use, because at 8 bits the simple approach is already close to lossless.
• K marks the k-quants added to llama.cpp in 2023. Weights are grouped into super-blocks of 256, split into smaller blocks whose scales are themselves quantized, as described in the original k-quants pull request.
• S, M and L (small, medium, large) describe how many sensitive tensors get extra bits. Per that same pull request, Q4_K_M stores half of the attention value tensors and half of the feed-forward down-projection tensors at 6-bit, and everything else at 4-bit.
• IQ marks the i-quants, newer formats built for low bit rates. They work best with an importance matrix, a measurement of which weights matter most on a sample of calibration text.

The real bits per weight run higher than the names suggest, because the per-block scales take space and the mixes keep some tensors at higher precision. Here is what llama.cpp measured for Llama 3.1 8B:

Source: llama.cpp quantize README. Your model’s numbers will differ a little, but the ratios hold.

Importance matrices and dynamic quants

Two refinements show up in file names on Hugging Face. An importance matrix (“imatrix”) is measured by running the full-precision model over calibration text; llama.cpp’s quantizer then uses it to decide where rounding errors hurt least, and its docs recommend it for reducing quality loss. Unsloth’s Dynamic quants go a step further and pick a different quant type for each layer based on how sensitive it is (Unsloth docs). Both help most at 4 bits and below.

What you lose at each level

Three kinds of evidence point the same way: almost nothing down to 5 or 6 bits, a small loss at 4, a clearer loss at 3, and a steep drop at 2 and below.

Perplexity from the original k-quant work

Perplexity measures how surprised a model is by real text; lower is better. The k-quants pull request reported these numbers for the original LLaMA 7B:

The same pull request found that a 30B model at Q2_K had lower perplexity than a 13B model at full 16-bit precision. That 2023 result is the origin of the advice to prefer a bigger model at lower bits.

Benchmarks on a modern instruct model

A January 2026 study (arXiv 2601.14277) ran Llama 3.1 8B Instruct through a benchmark suite at each GGUF level:

Down to Q4_K_M the average moved by about a third of a point, and Q5_K_M even scored slightly above Q6_K, which tells you the differences at that level are within noise. Q3_K_M lost 1.4 points. Q3_K_S lost about 4, with math hit hardest: GSM8K fell from 77.63 to 68.31.

Vendor numbers for a larger model

Unsloth publishes MMLU scores for its own quants of Gemma 3 27B: 71.5 at BF16, 71.60 at Q8_0, 71.87 at Q6_K, 71.23 at Q4_K_M, 70.70 at Q3_K_M, 59.20 at IQ2_XXS and 48.10 at IQ1_M (Unsloth docs). These are a vendor’s numbers for its own product, so treat them with some caution, but the shape matches the independent results: flat down to 4 bits, mild at 3, and a cliff at 2 bits and below.

Averages can hide changed answers

Benchmark averages can stay flat while individual answers change. The paper Accuracy is Not All You Need found that compressed models can match the original’s accuracy while flipping a meaningful number of answers, with right-to-wrong and wrong-to-right changes canceling out. The authors suggest KL divergence, which compares the full probability distributions of the two models, as a better measure, and llama.cpp’s tools can report it. If your work needs exact output, like long code edits or strict tool calls, test the quant on your own tasks and step up a level if you see odd mistakes.

Bigger model at lower bits, or smaller model at higher bits?

Usually the bigger model, down to about 4 bits and often to 3. On a 24GB card, Qwen3.8 27B at Q5_K_M (about 21GB) is a better choice than Qwen3.5 9B at Q8_0 (about 12GB) for almost any task. Below 3 bits the losses grow fast, and 1-bit and 2-bit files are best kept for very large mixture-of-experts models you couldn’t run any other way. Unsloth, which publishes 1-bit quants, says in its docs that they should not be used for agentic work.

Quantization also sets speed. This table shows the memory each level needs for three popular mid-size models:

Model	Size	Q3_K_M	Q4_K_M	Q5_K_M	Q6_K	Q8_0
Qwen3.8 27B	27B	15 GB	18 GB	22 GB	25 GB	32 GB
Gemma 4 31B	31B	19 GB	22 GB	26 GB	30 GB	36 GB
Qwen3.6 35B-A3B	35B (3B active)	19 GB	24 GB	28 GB	32 GB	41 GB

On an RTX 5090, our estimates for Qwen3.8 27B run from about 98 tok/s at Q3_K_M to about 80 tok/s at Q4_K_M, 69 at Q5_K_M, 60 at Q6_K and 47 at Q8_0. That’s the other half of why Q4_K_M is the default. Going from Q8_0 to Q4_K_M buys about 70% more speed for a quality loss that is hard to measure, while going lower costs quality faster than it adds speed.

Beyond GGUF: MLX, MXFP4 and NVFP4

MLX on Apple silicon

Apple’s MLX framework has its own quantization. By default it stores weights at 4 bits in groups of 64, each group with its own scale and bias, and it supports 1 to 8 bits plus the mxfp4, mxfp8 and nvfp4 modes, according to the MLX documentation. With 16-bit scales and biases, 4-bit MLX works out to about 4.5 bits per weight, close to Q4_K_M. On a Mac, LM Studio can run both GGUF and MLX models, and either format at 4 bits is a sensible default.

MXFP4

MXFP4 stores each weight as a 4-bit floating point number and gives every block of 32 values a shared 8-bit power-of-two scale, for about 4.25 bits per weight. OpenAI released gpt-oss-20b and gpt-oss-120b with their mixture-of-experts weights in MXFP4 and the rest in BF16. Hugging Face’s gpt-oss announcement notes this lets the 120B model fit on a single 80GB GPU and the 20B model in 16GB. llama.cpp runs MXFP4 natively, so use the MXFP4 GGUF as released instead of requantizing it.

NVFP4

NVFP4 is NVIDIA’s 4-bit format for Blackwell GPUs. Like MXFP4 it stores 4-bit floating point values, but it shares an FP8 scale across every 16 values instead of 32 and adds a per-tensor FP32 scale, for about 4.5 bits per weight. NVIDIA’s NVFP4 write-up describes it as about 3.5 times smaller than FP16 and reports an accuracy loss of 1% or less against FP8 on DeepSeek-R1-0528. Blackwell cards like the RTX 50 series and the RTX PRO 6000 Blackwell have FP4 math in hardware; older cards like the RTX 3090 and 4090 do not. Runtime support varies, so check yours before downloading NVFP4 checkpoints.

Quantizing the KV cache

The KV cache, the memory that holds your context, is quantized separately from the weights. llama.cpp stores it at 16-bit by default; the -ctk and -ctv flags switch keys and values to q8_0 or q4_0 (llama-server docs). In Ollama, set OLLAMA_KV_CACHE_TYPE. Ollama’s FAQ says q8_0 uses about half the memory with usually no noticeable impact, while q4_0 uses a quarter with a loss that may be more noticeable at long context.

For Qwen3.8 27B at 128K tokens, that is the difference between about 8.6GB of cache at 16-bit, 4.3GB at 8-bit and 2.1GB at 4-bit. Our VRAM guide has more on how context adds up.

How to choose a quant

1. Find out what fits. Use the VRAM calculator or a model’s requirements page, like the one for Qwen3.8 27B.
2. Start at Q4_K_M for the largest model that fits with 10 to 15% of memory free.
3. Spend spare memory on context first if you work with long documents or code, then on Q5_K_M or Q6_K. Q8_0 makes sense when memory is plentiful.
4. Below 4 bits, use imatrix or dynamic quants, and only when that gets you a much larger model.
5. Use native formats as released. For gpt-oss, that means MXFP4.

In practice, on a 16GB card like the RTX 5060 Ti 16GB, you can choose between Qwen3.8 27B at Q3_K_M (about 15GB and tight, ~26 tok/s estimated) and Qwen3.5 9B at Q8_0 (about 12GB); try both on your own tasks. On a 24GB RTX 3090, Qwen3.8 27B at Q5_K_M (about 21GB, ~34 tok/s estimated) is the sweet spot. On a 32GB RTX 5090, Q6_K (about 24GB) leaves room for long context. For which card gets you to each of those tiers, see our GPU buying guide.