Model quantization takes a neural network's weights, stored as 32-bit or 16-bit floating point numbers, and rounds them down to 8-bit or 4-bit integers. The math gets less precise. The model gets four to eight times smaller. Most of the time, you cannot tell the difference in output quality. That trade is why every model you actually run on a laptop, phone, or single GPU has been quantized in some form.
This is not a niche optimization trick anymore. It is the default path from research checkpoint to production. A 70-billion-parameter model in full precision needs roughly 140GB of memory just to load. Quantized to 4-bit, it fits in under 40GB. That is the difference between needing a rack of data center GPUs and running the thing on a single consumer card. Understanding quantization is understanding why open-source AI became usable outside of labs with unlimited compute budgets.
A neural network is a pile of numbers called weights and activations. In full precision (FP32), each number takes 32 bits, or 4 bytes, to store. Multiply that by billions of parameters and you get file sizes measured in hundreds of gigabytes. Quantization compresses those numbers into a smaller number of bits, usually 8, 4, or even 2, by mapping the original range of floating-point values onto a smaller set of integers.
Here is the mechanism in plain terms. Say a layer's weights range from -3.5 to 3.2. Instead of storing every exact decimal value, quantization picks a scale factor and a zero-point, then represents each weight as an integer between, say, -128 and 127 (8-bit). To use the weight during inference, the hardware multiplies the stored integer by the scale factor to reconstruct an approximation of the original value. You lose precision. You do not lose the general shape of what the model learned.
The reason this works better than you would expect is that neural networks are absurdly overparameterized. Most weights contribute small, redundant nudges to the final output. Rounding them slightly does not break the model's reasoning, because no single weight was ever doing the heavy lifting alone. This is also why quantization degrades gracefully instead of catastrophically. Push it too far, though, and it does break. That is where 2-bit quantization schemes start producing garbage output on anything but the largest models.
Every format tradeoff comes down to bits per parameter. Fewer bits means smaller size and faster math, at the cost of precision.
| Format | Bits | Relative size | Typical use |
|---|---|---|---|
| FP32 | 32 | 1x (baseline) | Training, research checkpoints |
| FP16 / BF16 | 16 | 0.5x | Standard inference on modern GPUs |
| INT8 | 8 | 0.25x | Production inference, edge servers |
| INT4 | 4 | 0.125x | Consumer GPUs, local LLM apps |
FP16 is not really quantization in the aggressive sense. It is a halving, and it barely costs any accuracy because it still has enough range and precision to represent most weight distributions cleanly. INT8 is where things get interesting: you are now storing whole numbers instead of floating-point values, which changes the math hardware needs to run. INT4 is where the real space savings show up, and where you start needing smarter techniques to avoid quality collapse.
GGUF deserves a separate mention because it is not a precision level, it is a file format built specifically for running quantized models on consumer hardware through tools like llama.cpp. GGUF files bundle the model architecture, tokenizer, and quantized weights into one portable file, often labeled with tags like Q4_K_M or Q5_1. The letter-number combos describe the quantization scheme used per layer, and yes, the naming is genuinely confusing. Q4_K_M is a reasonable default for most people running a model locally: 4-bit weights with a smarter per-block scaling method that keeps quality closer to the original than naive 4-bit rounding.
The honest answer: less than most people assume, until you push too hard. Going from FP16 to INT8 typically costs under 1% on standard benchmarks. Going from FP16 to INT4 costs somewhere between 1% and 5%, depending on the model and the quantization method used. Below 4-bit, degradation accelerates fast, and by 2-bit you are usually looking at a noticeably dumber model that hallucinates more and loses coherence on longer outputs.
Where quantization hurts most is on tasks requiring precise numerical reasoning, exact code output, or long chains of logical steps. A model doing creative writing or casual conversation can tolerate a lot of weight rounding because there is no single correct answer to get wrong. A model doing arithmetic or generating exact syntax has far less room for error, and quantization noise compounds across a long generation.
There is also a size-dependent effect that surprises people: bigger models tolerate quantization better than smaller ones. A quantized 70B model often outperforms a full-precision 7B model on the same task, at similar total memory footprint. This is because the 70B model has more redundant capacity to absorb rounding error, while the 7B model has less slack to begin with. If you are choosing between a small model at full precision and a large model quantized down, pick the large quantized model almost every time.
There are two fundamentally different ways to get a quantized model, and they are not interchangeable.
Post-training quantization (PTQ) takes an already-trained model and quantizes it afterward, with no retraining involved. It is fast, cheap, and works on any existing checkpoint. Methods like GPTQ and AWQ are sophisticated versions of PTQ: instead of naively rounding every weight the same way, they analyze which weights matter most for the model's output and preserve those with extra care, while being more aggressive on weights that barely matter. This is why a GPTQ or AWQ 4-bit model usually beats a naive 4-bit round-off by a wide margin on the same base model.
Quantization-aware training (QAT) bakes the quantization into the training process itself. The model learns to be robust to the precision loss because it experiences rounding during training, not just at the end. QAT produces better results at aggressive bit depths, but it requires access to training infrastructure and data, which most people downloading an open model simply do not have. In practice, almost everyone outside of the original model creators is stuck with PTQ, and that is fine. Modern PTQ methods have closed most of the gap that used to justify the extra cost of QAT.
Stop treating quantization choice as an afterthought. It should be the first decision you make when picking how to run a model, because it determines whether the thing fits on your hardware at all.
The mistake I see constantly: people default to whatever quantization level a tutorial happened to use, without checking whether their task can tolerate it. A customer support bot doing casual Q&A can run at 4-bit all day with nobody noticing. A financial analysis tool generating exact figures should not be anywhere near 4-bit without serious evaluation first. Match the precision to the cost of being wrong, not to whatever fits in the GPU you happen to own.
Quantization is not a compromise you settle for. It is closer to a free lunch, up to a point, and that point is lower than most people expect. The skill worth having is knowing where the cliff is for your specific task, not just knowing that the cliff exists.
Slightly, and only past a certain threshold. Going from 16-bit to 8-bit precision costs less than 1% on most benchmarks, which is close to unnoticeable. Below 4-bit, quality drops off faster, especially on math, code, and long multi-step reasoning.
INT8 stores each weight as an 8-bit integer, giving 256 possible values per number and very small accuracy loss compared to full precision. INT4 uses only 16 possible values per weight, cutting memory use in half again but requiring smarter techniques like GPTQ or AWQ to avoid a noticeable quality drop.
No, GGUF is a file format, not a precision level. It packages a quantized model's weights, architecture, and tokenizer into one file so tools like llama.cpp can run it on consumer CPUs and GPUs, and individual GGUF files still specify a quantization scheme such as Q4_K_M inside them.
Choose the larger quantized model almost every time at similar memory footprint. Big models have more redundant capacity to absorb rounding error, so a quantized 70B model typically outperforms a full-precision 7B model that uses the same amount of memory.
Yes, through methods like QLoRA, which fine-tune a small set of adapter weights on top of a frozen quantized base model. This lets you customize a large model on a single consumer GPU without ever loading the full-precision version into memory.