Learn › Local AI Agents: Hermes 3 and the Model Context Protocol › Why Hermes 3 and MCP: The Local Agentic Stack

Why Hermes 3 and MCP: The Local Agentic Stack

Advanced 🕐 10 min Lesson 1 of 12
What you'll learn
  • Explain what Hermes 3 is, why it was designed for tool calling, and which size to run for a given hardware budget
  • Describe the three MCP primitives — tools, resources, and prompt templates — and what distinguishes MCP from a custom tool wrapper
  • Sketch the full eight-step pipeline from user request to final answer in a Hermes 3 + MCP local agent
  • Identify the prerequisites needed before starting the hands-on lessons

The Problem With Most AI Agent Setups

Building an AI agent that can actually do things — read files, search the web, query a database, call an API — requires two components: a language model that understands how to call tools, and a way to wire those tools to the model in a consistent, composable way.

Most tutorials solve this with OpenAI's function calling API and a stack of Python wrappers. This works, but it has a price: every request leaves your machine, every query is logged by the vendor, and every month brings an API bill that scales with usage. For sensitive workloads, internal tools, or simply the desire to run everything locally, this is a non-starter.

The combination covered in this track solves those problems. Hermes 3 provides strong tool-calling capability that runs entirely on your hardware. The Model Context Protocol provides a standard way to define and connect tools without tying yourself to any particular LLM vendor or framework.

What Hermes 3 Is

Hermes 3 is a family of open-weight language models published by NousResearch, fine-tuned from Meta's Llama 3.1 and 3.2 base models. The family currently includes:

  • Hermes-3-Llama-3.2-3B: 3 billion parameters, runs on hardware as modest as a laptop with 8 GB RAM. Practical for simple tool-calling tasks where speed matters more than reasoning depth.
  • Hermes-3-Llama-3.1-8B: 8 billion parameters, the recommended starting point for most users. Fits comfortably on a 16 GB GPU and handles multi-step tool use reliably.
  • Hermes-3-Llama-3.1-70B: 70 billion parameters, requires 48+ GB of VRAM or multi-GPU inference. Approaches frontier model quality on complex reasoning tasks.

All three use ChatML as the conversation format and add a custom tool-calling protocol built on XML tags. The model emits a <tool_call> block when it wants to call a function, reads back the result from a <tool_response> block, and then continues generating its reply. This format is not an add-on — it was baked in during fine-tuning, which is why Hermes 3 is significantly more reliable at tool use than a general-purpose model prompted to call tools.

Hermes 3 is available as GGUF quantised files on Hugging Face, which means it runs on Ollama without any extra setup. Pulling the model, pointing Ollama at it, and querying it via the OpenAI-compatible API is all you need.

What the Model Context Protocol Is

The Model Context Protocol (MCP) is an open standard published by Anthropic in November 2024 for connecting AI models to external tools and data sources. It is not a framework, not a hosted service, and not specific to any one LLM. It is a protocol — a specification for how a client (your agent code) communicates with a server (a program that exposes tools or data).

MCP defines three primitives that a server can expose:

  • Tools: executable functions the agent can call — read a file, run a web search, insert a database row
  • Resources: read-only data the agent can inspect — a file's contents, a database schema, an API response
  • Prompt templates: reusable instruction templates the host can inject into the conversation

The practical benefit of using MCP rather than writing custom tool wrappers is portability. An MCP server you write today works with Claude Desktop, with VS Code, with Cursor, and — after Lesson 9 of this track — with Hermes 3 running locally. The same server works across all hosts that speak the protocol. When the broader ecosystem of MCP servers grows (filesystem, GitHub, Brave Search, Postgres, and dozens of others are already published), your local agent gains access to all of them without rewriting any glue code.

Why These Two Together

Hermes 3 and MCP are independent components that solve different parts of the same problem. Hermes 3 is the brain — it understands requests in natural language and decides which tools to call and with what arguments. MCP is the nervous system — it provides the standardised channel through which tools are discovered, invoked, and their results returned.

Neither one alone is sufficient. A language model without a tool layer can only answer from its training data. A tool layer without a capable tool-calling model produces unreliable, poorly-parsed calls. The combination — Hermes 3 as the LLM, MCP as the tool protocol — is the smallest stack that gets you a production-quality local agent.

The Full Stack

At the end of this track, the complete pipeline looks like this:

  1. The user sends a request in natural language (from a terminal, a web UI, or a script)
  2. A Python agent loop queries each connected MCP server for its available tools, assembling a tool registry
  3. The agent formats the tool definitions into Hermes 3's system prompt using the <tools> format
  4. The agent sends the prompt to Ollama and reads the response
  5. If the response contains a <tool_call> block, the agent parses it and calls the right MCP server
  6. The MCP server executes the tool and returns a result
  7. The agent injects the result as a <tool_response> turn and calls Ollama again
  8. Steps 4–7 repeat until Hermes 3 produces a final natural-language answer with no tool call in it

Every component in this pipeline — Ollama, the MCP servers, the Python agent loop — runs on your machine. No network requests leave your local environment unless a tool explicitly makes one (such as a web fetch tool you chose to include).

What You Will Build

By Lesson 12, you will have a working local research assistant. It connects to three MCP servers:

  • The official filesystem MCP server — reads and writes files in a designated workspace directory
  • The official fetch MCP server — retrieves web pages by URL and returns their text content
  • A custom notes MCP server you build in this track — stores and retrieves structured research notes with timestamps

The agent can answer questions like "summarise the contents of my meeting notes folder," "fetch the latest changelog from this URL and save it as a note," and "what notes did I take about project X last week" — all running locally on Hermes 3.

Prerequisites

This track assumes you are comfortable with Python 3.11+ and understand basic LLM concepts (prompts, chat turns, tokens). You should have Ollama installed — if not, Track 63 (Local AI: Run Models on Your Own Machine) covers setup. You do not need prior experience with MCP, Hermes, or agent frameworks.

Key takeaways
  • Hermes 3 uses a baked-in XML tool-calling format — it was fine-tuned specifically for this, which is why it outperforms a general model prompted to call tools
  • MCP is a protocol, not a framework — an MCP server you write once works with any host that speaks the protocol, including Claude Desktop, VS Code, and your own agent loop
  • The 8B model is the recommended starting point: it fits on 16 GB VRAM and handles multi-step tool use reliably without the hardware cost of the 70B
  • Nothing in the pipeline requires a cloud connection — Ollama runs the model locally, and the MCP servers run as local processes on the same machine
Learn more every week Get 5 hand-picked prompts + new AI tools every Monday. Free.