Auxiliary Model Routing for Cost Optimization

Not Every Request Needs Your Best Model

When Hermes processes a message, the main chat model handles the primary reasoning -- the turn-by-turn conversation where the agent decides what to do and produces outputs. But alongside that, Hermes runs a set of side tasks that support the main conversation: analyzing images, compressing long contexts, extracting content from web pages, and summarizing tool outputs.

By default, all of these side tasks go to the same model as the main chat. If you are using Claude Opus 4.8 as your main model, every image analysis, every context compression, and every web extraction also costs Opus 4.8 token rates. For some of these tasks, that is significant overspend -- image analysis and web extraction often produce perfectly acceptable results from a model that costs a tenth of Opus's price.

Auxiliary model routing lets you break the link between the main model and the side tasks. You configure a cheaper model for each auxiliary task type, and those side requests go to the cheaper endpoint while the main conversation continues using Opus. The agent's outputs are the same; the bill is lower.

The Auxiliary Tasks You Can Route

Three main auxiliary task slots are configurable in config.yaml:

auxiliary.vision handles image analysis. When Hermes needs to interpret an image -- reading a screenshot, analyzing a chart, understanding a photo -- this model handles it. Vision tasks benefit from a model with strong image understanding but do not necessarily need the full reasoning capability of your main model.

auxiliary.compression handles context window management. When a conversation gets long enough to approach the context limit, Hermes uses this model to compress the older parts of the conversation into a summary, making room for more recent content. Compression is a summarization task -- any capable model can do it well.

auxiliary.web_extraction (sometimes listed as web_summarization) handles content extraction from web pages. When the agent fetches a URL and needs to extract the relevant information from potentially large HTML content, this model processes the raw page and returns the useful parts. This is another task that does not require your most capable model.

The Universal Configuration Pattern

Every auxiliary task slot uses the same three-field pattern:

auxiliary:
  compression:
    provider: openrouter
    model: openai/gpt-4o-mini
    timeout: 60

The provider field specifies which API service to use. Valid values include auto (same as main model), main (alias for auto in auxiliary context), openrouter, nous, anthropic, openai, gemini, and a dozen more.

The model field specifies the exact model to request from that provider. Use the provider's model identifier format -- for OpenRouter, this is provider/model-name like openai/gpt-4o-mini or google/gemini-2.5-flash.

The optional timeout field sets how long Hermes waits for this auxiliary task to complete before giving up. Auxiliary tasks should be fast -- if vision analysis takes 60 seconds, something is wrong. Setting an explicit timeout prevents slow auxiliary tasks from blocking the main conversation indefinitely.

A Cost-Optimized Configuration Example

Here is a configuration that uses an expensive main model but routes auxiliary work to cheaper alternatives:

model: anthropic/claude-opus-4
auxiliary:
  vision:
    provider: openrouter
    model: google/gemini-2.5-flash
    timeout: 120
  compression:
    provider: openrouter
    model: openai/gpt-4o-mini
    timeout: 60
    fallback_chain:
      - provider: nous
        model: deepseek/deepseek-chat
  web_extraction:
    provider: openrouter
    model: google/gemini-2.5-flash
    timeout: 90

The main conversation uses Claude Opus 4 for maximum quality. Image analysis and web extraction go to Gemini 2.5 Flash on OpenRouter -- a capable model at a fraction of Opus's cost. Context compression goes to GPT-4o Mini with a fallback to Nous's DeepSeek instance if OpenRouter is unavailable.

The cost math here can be significant. If 30% of your session's token cost comes from auxiliary tasks, routing those to a model that costs 10x less reduces your total bill by roughly 27%. For batch jobs processing thousands of documents, that compounds quickly.

Per-Task Fallback Chains

Each auxiliary task slot can have its own independent fallback chain. This is separate from the main provider fallback and operates specifically for that task type:

auxiliary:
  compression:
    provider: openrouter
    model: openai/gpt-4o-mini
    fallback_chain:
      - provider: nous
        model: deepseek/deepseek-chat
      - provider: main
        model: null

If gpt-4o-mini fails, compression falls back to the Nous Portal's DeepSeek instance. If that also fails, it falls back to the main model (using provider: main) as a last resort -- ensuring context compression always completes even if both cheap options are unavailable.

This granularity is valuable: if your primary provider for auxiliary tasks has an outage, the main conversation is not affected. The auxiliary task handles its own failover independently.

Using Custom Endpoints for Auxiliary Tasks

If you are running local inference (via Ollama, LM Studio, or a self-hosted model), you can point auxiliary tasks at your local endpoint using the base_url override:

auxiliary:
  compression:
    base_url: http://localhost:11434/v1
    model: llama3.2:3b
    api_key: local-key

When base_url is set, the provider field is ignored and Hermes sends the request directly to your custom endpoint using OpenAI-compatible format. This is how you route auxiliary tasks to a local model while keeping the main chat on a cloud provider -- getting the quality of a large model for reasoning while using a local model to handle lower-stakes side tasks for free.

Monitoring Auxiliary Model Performance

Auxiliary model routing only saves money if the cheaper model actually performs well enough on the side tasks. Monitor your sessions to confirm auxiliary tasks are completing correctly. Signs of problems: context compression producing poor summaries that confuse subsequent reasoning, vision analysis returning vague or incorrect descriptions, web extraction missing key information from fetched pages.

If you see quality problems, move the affected auxiliary task to a more capable model. The cost savings from auxiliary routing are real but not worth compromising main task quality. Start with a mid-tier model (Gemini Flash, GPT-4o Mini) and drop to a smaller model only after confirming the output quality is acceptable for your use case.