Quantization Trade-Offs for Local Open-Weight Models

When you decide to run an open-weight model on your own hardware, you quickly hit a wall: the memory requirements for full FP16 weights are punishing—a 70B model needs ~140 GB of VRAM, which is a datacenter GPU. Quantization is the escape hatch that makes local inference practical, but each method trades off perplexity, memory, and inference speed in subtly different ways. Choosing the wrong level or format can leave your GPU idle while you wait for CPU fallback, or worse, silently degrade output quality on your domain.

Why run models locally

Privacy is the most obvious driver. When you send a prompt to a third-party API, the prompt text—and often the generated output—leaves your machine. For proprietary code, medical records, or internal strategy documents, that’s a non-starter. Local inference keeps every token on your hardware.

Cost is another factor. Once you own the GPU, inference consumes only electricity. There’s no per-token pricing and no rate limits. For teams running hundreds of thousands of inference calls per day, the break-even point against API providers typically arrives within a few months. After that, each additional call is essentially free.

Then there’s latency and control. A local model responds in milliseconds after the first token, not in the hundreds-of-milliseconds you’d add for network round trips. You control stop conditions, sampling parameters, and model versioning—no sudden deprecations, no undocumented changes to the underlying engine. If you’re building an agent loop that requires deterministic tool calls, being able to pin a specific quantized checkpoint is invaluable.

Quantization methods overview

The four main quantization formats for open-weight models serve different hardware profiles and inference engines.

GPTQ applies post-training quantization using an approximate Hessian-based weight rounding. The calibration process measures the importance of each weight to the model’s loss, then rounds to lower precision in a way that minimises perplexity increase. GPTQ is GPU-first, accelerated by ExLlama and AutoGPTQ. It works well on NVIDIA Ampere and Ada Lovelace architectures, but support for older GPUs (Maxwell, Pascal) is unreliable because the kernels depend on specific CUDA features.

GGUF is the format developed alongside llama.cpp. Its killer feature is mixed quantization: a single file can represent weights at different precision levels per layer (e.g., Q4_K_M uses 4-bit for most layers but 6-bit for the output projection). This lets you run reasonably large models on CPU or Apple Silicon, scaling context size without requiring more VRAM. GGUF is the default for Ollama, LM Studio, and most consumer-friendly local model runners.

AWQ (Activation-Aware Weight Quantization) quantizes weights based on the activation distribution of a calibration dataset rather than just weight statistics. The idea is that not all weights are equally important for activation ranges; AWQ protects the ones that handle out‑of‑range activations. In practice, AWQ often yields a small perplexity improvement over GPTQ at the same bit width (roughly 0.1–0.3 points on common benchmarks). It’s supported by vLLM and Text Generation Inference (TGI).

bitsandbytes is a library integrated with Hugging Face Transformers for naive 4‑bit and 8‑bit quantization. It’s convenient for prototyping: you pass load_in_4bit=True to from_pretrained() and you’re done. The trade-off is speed—bitsandbytes kernels are slower than the dedicated ones in ExLlama or llama.cpp, especially at batch size 1. Use it for quick experiments, but don’t deploy it for production inference.

Perplexity, memory, and speed: the core trade-off

Lower bit width reduces memory footprint linearly: 4‑bit weights occupy roughly half the memory of 8‑bit weights, and a quarter of FP16. But you pay for that saving in perplexity. The degradation depends on the method, the model size, and the calibration data.

Memory and speed figures are approximate and depend on hardware, batch size, and model architecture.

Inference speed is format-dependent. GPTQ and AWQ on GPU are typically fastest because their kernels are written in CUDA C++ and tuned for tensor cores. GGUF on GPU is slower because llama.cpp’s GPU offloading has some CPU fallback overhead, especially at layer boundaries. On CPU alone, GGUF is viable for mid-size models (7B–13B) but becomes sluggish for 70B models even with high‑bandwidth memory (DDR5).

The practical win: quantizing a 70B model to 4‑bit brings it from impossible (needing an A100) to runnable on a consumer 24 GB card like an RTX 4090 or 3090. That’s a 4× memory reduction for a small perplexity hit.

Choosing a quantization level for your hardware

Your hardware dictates the format, not the other way around.

GPU-only inference – If you have an NVIDIA GPU with at least 8 GB VRAM and CUDA compute capability 7.0 or higher (Volta, Turing, Ampere, Ada Lovelace, or Blackwell), use GPTQ or AWQ at 4‑bit. AWQ tends to give slightly better perplexity, but GPTQ has broader kernel support and faster development. Test both if you can. Avoid bitsandbytes unless you’re iterating on the code.

CPU or hybrid inference – If you don’t have a GPU or want to use very large context windows without VRAM bottlenecks, GGUF is the only practical choice. Q4_K_M is the sweet spot for quality vs. speed; Q5_K_M is worth it if you have the memory headroom. For low-memory devices like a MacBook with 8 GB unified memory, Q3_K_S may be necessary—but test perplexity carefully because the degradation is often steep.

Edge cases – Devices with less than 8 GB of system RAM (e.g., older laptops) struggle to load even a 7B model at 4‑bit. Here, Q2_K GGUF is the limit. The perplexity increase is noticeable (3–5%), but the model still works for simple tasks like summarisation or classification. If you’re evaluating LLM output on a dataset without a golden standard, make sure the quantization level doesn’t introduce systematic bias.

Practical pitfalls and gotchas

Incompatible kernels: GPTQ’s fused kernels are compiled for specific CUDA architectures. Trying to run a GPTQ model on a Maxwell GPU (compute capability 5.x) will either silently fall back to a slow CPU path or raise a cryptic CUDA error. Always check your GPU’s compute capability against the quantization library’s minimum requirements. AWQ has similar constraints, though its kernel coverage is growing.

Calibration data mismatch: GPTQ and AWQ quantization require a calibration dataset—commonly 128 sequences from WikiText‑2. If your production domain is code, legal text, or medical transcripts, the weight rounding decisions optimised for WikiText‑2 may be suboptimal. Perplexity on your own data could be 2–5% worse than published benchmarks. Always re‑evaluate on your own held‑out set.

Model-specific quirks: Mixture of Experts (MoE) architectures like Mixtral require special handling in GGUF because the expert routing weights are sensitive to quantization. Some early GGUF builds treated all layers uniformly, causing quality collapse. Current llama.cpp versions handle MoE correctly, but if you’re using a custom build, verify the implementation. Similarly, models with LoRA adapters stitched onto quantized bases sometimes fail if the adapter was trained at FP16—you may need to cast adapters.

Inference reproducibility: Quantized models have reduced numerical precision. The same prompt can produce slightly different token probabilities across runs due to nondeterministic CUDA kernels or floating-point rounding. If deterministic output is critical—for example, when validating an agent loop’s tool call chain—set a fixed seed in the inference engine and disable CUDA kernels that use atomic operations (e.g., torch.use_deterministic_algorithms(True)).

Benchmarking your own setup

Published benchmarks are useful as a sanity check, but your domain and hardware will shift the numbers. Run your own measurement.

Perplexity: Take 1000 samples from your own corpus (e.g., internal docs, support tickets). Compute cross‑entropy loss using the same tokenizer across FP16, GPTQ 4‑bit, and GGUF Q4_K_M. This tells you exactly how much quantization harms your use case.

Metrics: Profile tokens per second and peak GPU memory using nvidia-smi or torch.cuda.memory_summary(). For CPU inference, llama.cpp prints timing info to stderr. Run at least 50 prompts to capture variance.

Example: loading a GPTQ model with AutoGPTQ:

from auto_gptq import AutoGPTQForCausalLM
from transformers import AutoTokenizer
 
model_name = "TheBloke/Llama-2-7B-GPTQ"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device="cuda:0",
    use_triton=False,
    disable_exllama=False,
    use_safetensors=True,
)
 
input_ids = tokenizer("The capital of France is", return_tensors="pt").input_ids.to("cuda")
output = model.generate(input_ids, max_new_tokens=50)
print(tokenizer.decode(output[0]))

For GGUF with llama.cpp Python bindings:

from llama_cpp import Llama
 
llm = Llama(
    model_path="./models/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf",
    n_ctx=4096,
    n_threads=8,
    n_gpu_layers=-1,  # Offload all layers to GPU if available
)
output = llm("What is the capital of France?", max_tokens=50)
print(output["choices"][0]["text"])

Don’t trust single‑sentence demos. A model that answers “Paris” correctly may still fail on multi‑step reasoning. Run a batch of diverse prompts to see the spread.

A decision flow for picking quantization

Start with the largest model that fits your VRAM budget at 4‑bit. If that’s a 70B model on a 24 GB card, use GPTQ or AWQ. If even that doesn’t fit, drop to 3‑bit—but evaluate perplexity carefully; 3‑bit GGUF is often too noisy for factual recall. If you’re on CPU, GGUF Q4_K_M on the largest model that fits in system RAM.

Choose the format that your inference engine supports natively. If you’re using Ollama or llama.cpp, pick GGUF. If you’re using TGI or vLLM, pick AWQ or GPTQ. Avoid mixing formats unless you enjoy debugging C++ ABI mismatches.

Once you’ve settled on a quantization level, pin the versions of the quantizer (e.g., AutoGPTQ 0.7.0) and the inference runtime (llama.cpp commit hash). Quantization code changes rapidly; a library upgrade can silently alter weight rounding behaviour and change your model’s output.

Key takeaways

• Quantization reduces memory by 4× (FP16 → 4‑bit) at the cost of a small perplexity increase (0.5–3% depending on method).
• GPTQ and AWQ are the fastest options for GPU inference; GGUF is the only practical choice for CPU and Apple Silicon.
• Calibration data mismatch is the most common cause of unexpected quality loss—always benchmark on your own domain.
• Pin library versions after you choose a quantization level; updates can change rounding behaviour and break reproducibility.
• For production agent loops or structured output pipelines, run deterministic inference with a fixed seed to avoid token‑level variance.