What Is Quantization?

Neural networks store weights as numerical values. By default, these values use 32-bit floating-point numbers (FP32). Quantization reduces the precision of these numbers—using fewer bits to represent each weight.

The result:

• Smaller models — less memory required
• Faster inference — less data to read from memory
• Lower quality — some precision is lost

The art of quantization is finding the balance where memory and speed gains outweigh quality loss.

The Four Main Formats

FP32: Full Precision

32-bit floating point is the standard format for training and storing neural network weights.

FP16: Half Precision

16-bit floating point halves the memory requirement while maintaining nearly identical quality.

FP16 is the standard inference format. Most GPU-accelerated inference uses FP16 or BF16 (brain floating point, which has the same memory but different exponent range).

INT8: 8-bit Integer

8-bit integer quantization maps floating-point weights to a limited range of integer values.

INT8 uses a scaling factor to map the original floating-point range to integer values. The weight is stored as an integer, and during inference, it's scaled back:

# Conceptual INT8 quantization
original_weight = 0.23456789
scale_factor = 0.002  # Determined during calibration
quantized_weight = round(original_weight / scale_factor)  # = 117
# During inference:
restored_weight = quantized_weight * scale_factor  # = 0.234

INT4: 4-bit Integer

4-bit integer quantization achieves the highest compression but with more quality loss.

Modern INT4 quantization methods like Q4_K_M use advanced techniques to minimize quality loss:

• Block quantization: Different scales for different weight groups
• K-means clustering: Optimal value selection
• Super-blocks: Hierarchical scaling
• Mixed precision: Critical layers get higher precision

Numerical Demonstration

Let's trace a single weight through all four formats:

Step 1: FP32 Representation

# FP32: Full precision (32 bits)
weight_fp32 = 0.23456789111328125
# Bits: 0 01111101 11101110010101100000000
#       ^ ^--------^ ^---------------------^
#       | exponent   | mantissa (23 bits)
#       sign (1 bit)
# 
# Memory per weight: 4 bytes
# 7B model total: 28 GB

Step 2: FP16 Representation

# FP16: Half precision (16 bits)
weight_fp16 = 0.23450
# Bits: 0 01111 1101011100
#       ^ ^-----^ ^-------^
#       | exponent| mantissa (10 bits)
#       sign (1 bit)
#
# Error: |0.23456789 - 0.23450| = 0.00006789 (0.029%)
# Memory per weight: 2 bytes
# 7B model total: 14 GB

Step 3: INT8 Representation

# INT8: 8-bit integer with scaling
# Scale factor: 0.00183 (determined during calibration)
scale = 0.00183
zero_point = 0

# Quantize
weight_int8 = round(0.23456789 / 0.00183)  # = 128
# Clamp to valid range [0, 255]
weight_int8 = min(128, 255)  # = 128

# Dequantize
weight_restored = 128 * 0.00183  # = 0.23424

# Error: |0.23456789 - 0.23424| = 0.00032789 (0.14%)
# Memory per weight: 1 byte
# 7B model total: 7 GB

Step 4: INT4 Representation

# INT4: 4-bit integer with block scaling
# Block size: 32 weights share one scale
block_scale = 0.0156

# Quantize
weight_int4 = round(0.23456789 / 0.0156)  # = 15
# Clamp to valid range [0, 15]
weight_int4 = min(15, 15)  # = 15

# Dequantize
weight_restored = 15 * 0.0156  # = 0.234

# Error: |0.23456789 - 0.234| = 0.00056789 (0.24%)
# Memory per weight: 0.5 bytes
# 7B model total: 3.5 GB

Summary: Single Weight Comparison

Memory Impact: 7B Model

Let's see how quantization affects an entire 7B parameter model:

Adding context length (KV cache) further affects memory:

Speed Impact

CPU inference speed is primarily limited by memory bandwidth. Smaller models mean less data to read, so quantization directly improves speed:

Quality Impact

Quality loss depends on the quantization method, not just the bit width. Modern methods like GPTQ, AWQ, and GGUF handle quantization much better than naive approaches.

Where Quality Loss Matters Most

• Mechanical tasks (math, code syntax) — less affected
• Reasoning tasks — moderate impact
• Creative writing — slight style changes
• Factual recall — can degrade with heavy quantization
• Long-context tasks — more sensitive to precision

Advanced Quantization Techniques

Block Quantization

Instead of a single scale for the entire model, weights are divided into blocks. Each block has its own scale factor.

# Block quantization example
# Block size: 32 weights

# Block 1 weights: [0.1, 0.3, 0.5, 0.2, ...]
block1_scale = 0.02
block1_int4 = [5, 15, 25, 10, ...]  # Quantized

# Block 2 weights: [-0.4, -0.2, 0.1, 0.6, ...]
block2_scale = 0.03
block2_int4 = [-13, -7, 3, 20, ...]  # Quantized

Group Quantization

Groups of weights share quantization parameters, providing better accuracy than single-weight quantization.

# Group quantization
# Group size: 128 weights
# Each group has: scale + zero_point + 128 quantized weights

group = {
    "scale": 0.015,
    "zero_point": 8,
    "weights": [3, 12, 7, 15, 2, ...]  # 128 values
}

Mixed Precision

Critical layers (attention heads, output layers) can use higher precision while less sensitive layers use lower precision.

# Mixed precision example
model_layers = {
    "embeddings": "FP16",      # Sensitive to precision
    "attention": "INT8",       # Moderate sensitivity
    "ffn": "INT4",             # Less sensitive
    "layer_norm": "FP16",      # Critical for stability
    "output_head": "INT8",     # Affects final output
}

Choosing the Right Quantization

Quantization Quality Checklist

Common Quantization Myths

Conclusion

Quantization is essential for running LLMs on consumer hardware. The key points:

1. FP16 is the standard for GPU inference with negligible quality loss
2. INT8 provides 2x memory savings with minimal quality impact
3. INT4 (Q4_K_M) offers 4x compression while retaining 95-97% quality
4. Memory bandwidth is the primary speed factor, not CPU/GPU compute
5. Always test quantization quality on your specific use case

The choice between formats depends on your hardware, use case, and quality requirements. For most users running models locally, Q4_K_M provides the best balance of memory, speed, and quality.

Further Reading

• Running LLMs on CPU: What Actually Matters?
• Local AI in 2026: What Can You Really Run on a Laptop?
• Ollama vs llama.cpp vs LM Studio
• Build a Private Local AI Agent with MCP
• BestWordz Hash Generator - Useful for verifying model files

Discuss This Topic

Have questions about quantization? Join the conversation on BestWordz Community.