Running Hermes 3 with Ollama

Advanced 🕐 10 min Lesson 7 of 12
What you'll learn
  • Pull the Hermes 3 8B GGUF model from Hugging Face via Ollama and verify it appears in ollama list
  • Create a Modelfile alias with num_ctx and temperature set for tool calling
  • Call the Ollama OpenAI-compatible API from Python using requests and get a successful response
  • Explain why context window size matters for tool calling and how to adjust it in the Modelfile

Assumptions

This lesson assumes Ollama is already installed and running. If it is not, Track 63 (Local AI: Run Models on Your Own Machine) covers the full Ollama setup including installation, the Modelfile system, and the API. This lesson starts from "Ollama works" and covers only the Hermes-specific steps.

Pulling the Hermes 3 Model

Hermes 3 is available on Ollama's model library. Pull the 8B variant — it is the recommended starting point for tool use on 16 GB VRAM:

ollama pull hf.co/NousResearch/Hermes-3-Llama-3.1-8B-GGUF:Q4_K_M

This pulls the 4-bit quantised GGUF file directly from NousResearch's Hugging Face repository. The Q4_K_M variant is approximately 5 GB and runs on 8 GB VRAM, though generation is faster with 16 GB.

Alternative sizes if 8B is too large or too slow for your hardware:

  • 3B (Q4_K_M, ~2 GB): ollama pull hf.co/NousResearch/Hermes-3-Llama-3.2-3B-GGUF:Q4_K_M — runs on 4–6 GB VRAM, adequate for simple single-tool calls
  • 8B (Q8_0, ~9 GB): Replace Q4_K_M with Q8_0 for near-full quality; requires 12+ GB VRAM
  • 70B (Q4_K_M, ~42 GB): ollama pull hf.co/NousResearch/Hermes-3-Llama-3.1-70B-GGUF:Q4_K_M — requires 48+ GB VRAM or multi-GPU

Wait for the pull to complete, then verify it is registered:

ollama list

The model name shown in ollama list is the exact name you will use in your agent code when calling the API.

Creating a Local Alias with a Modelfile

The Hugging Face model path is long to type. Create a shorter local alias using a Modelfile:

FROM hf.co/NousResearch/Hermes-3-Llama-3.1-8B-GGUF:Q4_K_M

PARAMETER num_ctx 8192
PARAMETER temperature 0.1
SYSTEM "You are a helpful AI assistant with access to tools. Use them when needed to answer the user accurately."

Save this as Modelfile.hermes, then create the alias:

ollama create hermes3 -f Modelfile.hermes

Now you can refer to the model as hermes3 in your agent code. The num_ctx 8192 parameter sets the context window to 8,192 tokens — important for tool calling, which consumes significant context with tool definitions in the system prompt. The temperature 0.1 keeps output deterministic for structured tool calls.

Important: the system prompt in the Modelfile is a base persona only. The agent code will inject its own system prompt containing the tool definitions — this base prompt does not need to list tools. The agent's system prompt overrides or extends this base depending on how it is constructed.

Verifying the Model Runs

Test the model with a simple chat before using it in the agent:

ollama run hermes3 "What is 2 + 2?"

You should get a coherent response within a few seconds. If the model does not load, check that enough VRAM is available: nvidia-smi on Linux/Windows or Activity Monitor → GPU History on macOS shows current VRAM usage.

The Ollama API

Ollama exposes an OpenAI-compatible HTTP API at http://localhost:11434/v1. Your agent code uses this endpoint directly — no Ollama Python library needed beyond the standard requests or httpx library, though the openai Python SDK also works because Ollama speaks the same API format.

A minimal test using the Python requests library:

import requests, json

response = requests.post(
    "http://localhost:11434/v1/chat/completions",
    json={
        "model": "hermes3",
        "messages": [
            {"role": "user", "content": "What is 2 + 2?"}
        ],
        "temperature": 0.1,
        "stream": False,
    }
)
data = response.json()
print(data["choices"][0]["message"]["content"])

Run this and confirm you get a response. This is the same call your agent loop will make, except the system message will contain the Hermes tool definitions.

Context Window and Tool Overhead

Tool calling consumes context. The system prompt for a Hermes 3 agent includes:

  • The Hermes preamble and FunctionCall schema (roughly 200 tokens)
  • Each tool definition: name, description, and parameter schema (roughly 50–150 tokens per tool)

With 10 tools across three servers, the system prompt alone uses roughly 1,500 tokens. Add the conversation history and the remaining context is what Hermes has for generating responses. The num_ctx 8192 setting leaves about 6,500 tokens for conversation history after tool definitions — enough for most research tasks.

If you find the model running out of context mid-task, increase num_ctx in the Modelfile. The 8B Q4_K_M model can handle up to 32,768 tokens on 16 GB VRAM, though generation slows at larger context windows. Update the Modelfile and run ollama create hermes3 -f Modelfile.hermes again to apply the change.

Keeping Ollama Running

Ollama runs as a background service. It starts automatically on most systems after installation. If the API is not reachable, start it manually:

ollama serve

On Linux with systemd: systemctl start ollama. The service must be running before your agent code makes any API calls. If the agent gets a connection error on startup, this is the first thing to check.

Key takeaways
  • The Hugging Face model path is the exact string passed to ollama pull — create a Modelfile alias so agent code uses a short name like hermes3 instead of the full hf.co path
  • Set temperature to 0.1 in the Modelfile for tool calling — lower temperatures produce more deterministic structured output, which matters for JSON tool call arguments
  • num_ctx 8192 is a reasonable default; tool definitions consume 50-150 tokens each, so 10 tools across three servers uses roughly 1500 context tokens before any conversation history
  • The Ollama API is OpenAI-compatible at http://localhost:11434/v1 — the standard openai Python SDK works against it by setting base_url and api_key to any non-empty string
Learn more every week Get 5 hand-picked prompts + new AI tools every Monday. Free.