A 7 billion parameter model in full precision needs 28 GB of RAM. Most laptops have 8-16 GB. Without quantization, local AI on consumer hardware is impossible for anything beyond the smallest models.
Quantization solves this by representing model weights with fewer bits. The same 7B model in 4-bit precision needs only 3.5 GB. That is the difference between "impossible" and "runs on your laptop."
What Is Numerical Precision?
Every number in a neural network is stored as a sequence of bits (0s and 1s). The more bits used, the more precisely the number can be represented. This is numerical precision.
| Format | Bits | Bytes | Range | Precision |
|---|---|---|---|---|
| FP32 | 32 | 4 | ±3.4×10³⁸ | ~7 decimal digits |
| FP16 | 16 | 2 | ±65,504 | ~3 decimal digits |
| BF16 | 16 | 2 | ±3.4×10³⁸ | ~2 decimal digits |
| INT8 | 8 | 1 | -128 to 127 | 256 levels |
| INT4 | 4 | 0.5 | -8 to 7 | 16 levels |
FP32: Full Precision
32-bit floating point. The standard format for training. Full range and precision. Requires 4 bytes per weight.
FP16: Half Precision
16-bit floating point. Half the size of FP32. Used for mixed-precision training and inference. Limited range (max ~65,500) but sufficient for most model weights.
BF16: Brain Float 16
16-bit format designed for deep learning. Same range as FP32 (±3.4×10³⁸) but less precision (~2 decimal digits). Preferred by some training frameworks because it avoids overflow issues.
INT8: 8-bit Integer
8-bit integer. 256 discrete levels (-128 to 127). Half the size of FP16. Uses a scaling factor to map between the integer range and the original float range.
INT4: 4-bit Integer
4-bit integer. Only 16 discrete levels (-8 to 7). One-quarter the size of INT8. This is where quantization becomes transformative for consumer hardware.
How Quantization Works: A Numerical Example
Here is what happens to a single model weight when quantized:
FP32 (32 bits): 0.4500000 → 4 bytes → exact representation
FP16 (16 bits): 0.4499512 → 2 bytes → tiny error (0.00005)
INT8 (8 bits): 0.4488189 → 1 byte → small error (0.001)
INT4 (4 bits): 0.4285714 → 0.5 bytes → larger error (0.021)
As bits decrease: size shrinks, precision drops, error grows
Memory Impact: Why It Matters
Model size is determined by: parameters × bits_per_weight / 8
| Precision | Bits | 7B Model | 13B Model | RAM Needed |
|---|---|---|---|---|
| FP32 | 32 | 28 GB | 52 GB | 32+ GB |
| FP16 | 16 | 14 GB | 26 GB | 20+ GB |
| INT8 | 8 | 7 GB | 13 GB | 10+ GB |
| INT4 | 4 | 3.5 GB | 6.5 GB | 6+ GB |
A 7B model that needs 28 GB in FP32 fits in 3.5 GB in INT4 — that is the difference between "server only" and "runs on any laptop."
The Quantization Trade-off
↓
Quantization (reduce bit precision)
↓
Smaller Model (FP16 / INT8 / INT4)
↓
Lower Memory Requirement
↓
Potential Quality Trade-off (depends on model + task)
What You Gain
- Smaller model: 2× to 8× reduction in size
- Less RAM: Run models that would not fit otherwise
- Faster inference: Fewer bits = fewer calculations = faster
- Lower cost: Run on consumer hardware instead of servers
What You Might Lose
- Numerical precision: Weights are approximate, not exact
- Complex reasoning: Tasks requiring precise math may suffer
- Edge cases: Rare or unusual inputs may produce worse results
- Very small models: Less redundancy to absorb precision loss
Quality Impact: Where It Matters
| Task Type | Impact of INT4 | Explanation |
|---|---|---|
| Simple Q&A | Minimal | Core knowledge preserved |
| Summarization | Minimal | Pattern matching, not precision |
| Code generation | Low | Common patterns well-learned |
| Translation | Low | Language structure preserved |
| Complex reasoning | Moderate | Precision loss can compound |
| Mathematical calculation | High | Exact values matter |
| Precise recall | High | Memory of training data affected |
Quantization Methods
Not all quantization is equal. The method matters as much as the bit count.
| Method | Type | Quality | Speed | Used By |
|---|---|---|---|---|
| Q4_K_M | GGUF block | Good | Fast | Ollama, llama.cpp |
| Q5_K_M | GGUF block | Very Good | Medium | Ollama, llama.cpp |
| Q8_0 | GGUF block | Best | Slow | Ollama, llama.cpp |
| GPTQ | Layer-wise | Good | Fast (GPU) | Transformers |
| AWQ | Activation-aware | Very Good | Fast | Transformers |
| GGUF | File format | Varies | CPU+GPU | llama.cpp ecosystem |
GGUF and Local Inference
GGUF is the file format used by llama.cpp (and by extension, Ollama and LM Studio). It bundles quantized weights, tokenizer data, and model metadata into a single file. When you download a quantized model for local AI, you are almost always downloading a GGUF file.
📖 Read more: GGUF Explained: The Practical Guide to Local LLM Model FilesHow to Choose a Quantized Model
Use this decision framework to pick the right quantization for your situation:
| Your Situation | Recommended | Why |
|---|---|---|
| 8GB RAM, no GPU | Q4_K_M (3-7B) | Fits in RAM, good quality |
| 16GB RAM, no GPU | Q4_K_M (7-13B) | Best balance for most users |
| 16GB RAM + GPU | Q4_K_M or Q5_K_M | GPU accelerates, Q5 for better quality |
| 32GB RAM | Q5_K_M or Q8_0 | Enough RAM for higher precision |
| Quality is critical | Q8_0 or FP16 | Maximum precision available |
| Speed is critical | Q4_K_S or Q3_K_M | Fewer bits = faster inference |
| Exploring / testing | Q4_K_M | Standard default, good for everything |
Common Quantization Myths
| Myth | Reality |
|---|---|
| "4-bit is always the same quality as FP16" | It depends on the model, method, and task. Often close, but not identical. |
| "Quantization always makes models worse" | For many tasks, the quality difference is negligible. For some tasks, it is noticeable. |
| "Lower bits always means faster" | Not always. Some quantization methods add overhead. GGUF Q4 is generally faster. |
| "You should always use the lowest quantization" | Balance quality with your RAM. Q4_K_M is usually the sweet spot. |
| "Quantization is lossless" | It is lossy. Information is lost. The question is whether that loss matters for your task. |
FAQ
Q: What quantization should I use?
A: Q4_K_M for most use cases. It provides the best balance of quality, speed, and memory for local inference.
Q: Can I run a 7B model on 8GB RAM?
A: Yes, with Q4_K_M quantization (3.5 GB). Leave room for the OS and runtime (~2-3 GB).
Q: Does quantization affect speed?
A: Yes. Fewer bits means fewer calculations, which is generally faster. INT4 inference is typically 2-3× faster than FP32.
Q: What is GGUF?
A: A file format for storing quantized model weights, used by llama.cpp, Ollama, and LM Studio. It bundles weights, tokenizer, and metadata.
Q: Can I quantize any model?
A: Most models can be quantized. GGUF quantization is done by tools like llama-quantize. GPTQ and AWQ have their own pipelines.
What to Learn Next
Further Reading
- Local AI Explained
- GGUF Explained
- Ollama Tutorial
- llama.cpp Explained
- LM Studio Tutorial
- Local AI in 2026
Continue Learning: Understand what local AI is, learn about GGUF format, choose your runtime, and check what your hardware can run.
Discuss this topic on BestWordz Community.