Understanding Models: GGUF, MLX, and Quantization

What Is a Model File?

When you download a local AI model, you are downloading a single large file (or a small set of files) that contains billions of numerical weights — the learned parameters that make the model work. A 7B model has around 7 billion of these weights. At full 32-bit precision, that would be 28 GB. In practice, you never download the raw research format — you download a compressed, inference-optimised version designed to run efficiently on consumer hardware.

Two formats dominate local AI in 2026: GGUF and MLX.

GGUF: The Universal Format

GGUF (GPT-Generated Unified Format) is the standard format for running models on CPU and GPU across all platforms — Windows, Linux, and macOS. It is the format used by llama.cpp, the inference engine that powers both Ollama and LM Studio on non-Apple hardware.

GGUF files are hosted on Hugging Face and named to tell you exactly what is inside. A file named llama-3.3-8B-instruct-Q4_K_M.gguf breaks down like this:

• llama-3.3 — the model family and version
• 8B — 8 billion parameters
• instruct — fine-tuned for following instructions (as opposed to a raw base model)
• Q4_K_M — quantization level (explained below)

GGUF works on any hardware that supports llama.cpp — NVIDIA GPUs, AMD GPUs, Apple Metal, and CPU-only. It is the safest choice if you are unsure which format to pick.

MLX: Apple Silicon Only

MLX is Apple’s own machine learning framework, optimised for the unified memory architecture of M-series chips. LM Studio can download models in MLX format directly, and as of 2026 MLX inference is 2–2.5x faster than llama.cpp on the same Apple Silicon hardware.

If you are on an M1, M2, M3, M4, or M5 Mac, always prefer MLX format when LM Studio offers it. The quality is identical to GGUF at the same quantization level — the speed difference is purely from hardware optimisation. Ollama uses Metal GPU acceleration natively on Apple Silicon without needing you to pick the MLX format explicitly.

Quantization: Trading a Little Quality for a Lot of Speed

Quantization reduces the precision of each weight from 32-bit floating point down to fewer bits. This makes the file smaller and inference faster at the cost of a small amount of accuracy. Here is the practical meaning of the common levels:

• Q2: Very aggressive compression. File is tiny but quality degrades noticeably. Only worth using if VRAM is extremely limited.
• Q4 (Q4_K_M recommended): The sweet spot. File is roughly half the size of Q8 with only a 1–3% quality drop on most benchmarks. Q4_K_M is the variant most people recommend — it uses a mixed-precision strategy that preserves accuracy on key layers. This is what you should start with.
• Q8: Near-identical to full precision. About 8 GB for a 7B model. Worth using when VRAM is plentiful and you want the best possible quality.

For a 7B model, Q4_K_M is typically around 4.5 GB. Q8 is around 8 GB. On a 8 GB VRAM card, Q4_K_M fits entirely in VRAM; Q8 would require partial CPU offloading and run slower. For most users, Q4_K_M is the right default.

How This Appears in Practice

When you run ollama pull llama3.3, Ollama automatically downloads the recommended quantization for your hardware. In LM Studio, the model browser shows GGUF and MLX options with file sizes next to each quantization level — you can see exactly what you are downloading and compare the size-to-quality tradeoff before committing to a download.

You do not need to memorise all quantization levels. The decision is usually: does Q4_K_M fit in my VRAM? If yes, use it. If you have headroom, consider Q8. If Q4 does not fit, the model is too large for your hardware and you should pick a smaller one rather than forcing a lower quantization.