Choosing Your Model for Your Hardware

The Biggest Mistake Beginners Make

When people discover they can run a 70B model on their machine — technically, with enough disk space and RAM — they try to. The result is a model that runs at 1–3 tokens per second, loses conversation history because the context window is constantly overflowing, and feels far worse than the 8B model would have felt at 40 tokens per second.

The rule is simple: run the largest model that fits entirely in your VRAM, not the largest model your machine can technically hold. When a model spills over from GPU memory into system RAM, inference slows by 5–10x. A fast 8B model in VRAM will almost always feel better than a slow 27B model split across VRAM and RAM.

A secondary trap is memory bandwidth. Two graphics cards with identical VRAM capacity can perform very differently for local AI. VRAM capacity determines what fits; memory bandwidth determines how fast tokens generate. An RTX 4090 generates tokens faster than an RTX 3090 not just because it has more VRAM, but because its memory bandwidth is 1,008 GB/s vs 936 GB/s. This is why Apple Silicon punches above its weight — unified memory bandwidth on an M3 Max reaches 300 GB/s, which is competitive with many desktop GPUs for model inference.

The 2026 Model Landscape

These are the model families most commonly used for local AI in 2026, with real hardware requirements and token speeds from tested benchmarks:

Llama 3.3 (Meta)

The most widely used open-weight model family. Llama 3.3 8B requires 6 GB VRAM at Q4 and delivers around 40 tokens per second on an RTX 4080 — fast enough for a natural conversation. It handles most daily tasks: summarisation, coding assistance, Q&A, and general writing. Llama 3.3 70B demands 40+ GB VRAM but delivers significantly stronger reasoning for complex tasks.

Run in Ollama: ollama run llama3.3

Mistral Small 3 (Mistral AI)

The fastest model at inference — around 50 tokens per second on a 16 GB VRAM card. Excellent for tasks where speed matters more than depth: quick rewrites, short summaries, rapid code completion. Particularly strong for European languages.

Run in Ollama: ollama run mistral-small3

Gemma 3 (Google)

Gemma 3 1B is extraordinarily compact at 0.5–2 GB and runs on CPU-only hardware. It is genuinely usable for simple tasks on a laptop without a dedicated GPU. Gemma 3 27B is multimodal — it can process both text and images — and excels at multilingual tasks, supporting over 140 languages.

Run in Ollama: ollama run gemma3 (downloads the 4B default) or ollama run gemma3:27b

Phi-4-mini (Microsoft)

At 3.8B parameters and 3.5 GB VRAM at Q4, Phi-4-mini is the best choice for hardware with 8 GB RAM and no dedicated GPU. Microsoft trained it specifically to punch above its weight on reasoning tasks. It scores higher on coding benchmarks than many larger models and is the recommended starting point for laptop users without a GPU.

Run in Ollama: ollama run phi4-mini

Qwen3 (Alibaba)

The most actively developed model family in 2026. Qwen3 7B posts the best HumanEval score in its parameter class (76.0 for coding tasks) and supports over 90 languages, making it the top choice for multilingual work or coding assistance on mid-range hardware. Qwen3 32B (22 GB VRAM) offers an excellent quality-to-size ratio for users with higher-end hardware.

Run in Ollama: ollama run qwen3

Matching Models to Tasks

No single model is best at everything. Use this as a rough guide:

• Coding assistance: Qwen3 7B or Phi-4-mini for speed; Llama 3.3 70B if you have the hardware
• General chat and summarisation: Llama 3.3 8B — the well-rounded default
• Fast responses, speed-critical tasks: Mistral Small 3 7B
• No GPU / CPU-only machines: Phi-4-mini or Gemma 3 1B
• Multilingual work: Qwen3 (90+ languages) or Gemma 3 27B (140+ languages)
• Image analysis alongside text: Gemma 3 27B, Llama 3.2 Vision, or Qwen2.5-VL

A Practical First-Run Recommendation

If you are not sure where to start, run ollama run llama3.3 on any machine with 8+ GB VRAM, or ollama run phi4-mini on a laptop. Both download in under 5 minutes and give you a solid baseline to compare everything else against.