Cost Architecture: Provider Routing, Auxiliary Models, and Credential Pools

The Multi-Model Reality

A production Hermes deployment rarely uses a single model for everything. The primary model handles complex reasoning. A cheaper, faster model handles summarization, session titles, and memory management. A vision-capable model handles image inputs. A background review model evaluates output quality without consuming the primary model's capacity. Using one model for all of this is expensive and often slower than it needs to be.

Hermes has a layered model configuration system designed for exactly this reality: provider routing for fine-grained control over which models handle primary requests, auxiliary model slots for side tasks, fallback providers for resilience, and credential pools for distribution. Understanding this system is what separates a Hermes deployment that burns $60/month on API calls from one that accomplishes the same work for $12.

Provider Routing

Provider routing applies when you are using OpenRouter or Nous Portal as your model provider. These platforms route your requests to the actual underlying models (Anthropic, OpenAI, Mistral, etc.) and give you control over how that routing works.

The routing configuration lives under your model configuration in config.yaml and accepts six parameters:

• sort — rank providers by price, throughput, or latency. sort: price routes to the cheapest available provider for the requested model. sort: throughput routes to the one with the highest tokens-per-second rate. sort: latency routes to the one with the lowest time-to-first-token.
• only — whitelist specific providers. If you have data sovereignty requirements that prevent using certain cloud providers, only: [anthropic] ensures your requests only go to Anthropic's infrastructure via OpenRouter, regardless of pricing or availability on other providers.
• ignore — blacklist specific providers. If you have had reliability issues with a specific provider or have rate-limit agreements that make a provider expensive for your use case, exclude it.
• order — set an explicit priority sequence. The first provider in the list is tried first; unlisted providers are used as fallbacks. Useful when you have a preferred provider for quality reasons but want a fallback for availability.
• require_parameters — route only to providers that support all the parameters in your request. If you are using tool use with specific parameters that not all providers support, this ensures you only land on a compatible provider.
• data_collection — set to allow or deny to control whether your requests can be used by providers for training data. For enterprise use with proprietary codebases, deny is the correct setting.

Auxiliary Models

Hermes uses separate model slots for several categories of side work. Each slot can be configured independently of the primary model, allowing you to route lightweight tasks to cheap, fast models while reserving the primary model for reasoning-heavy turns.

The configurable auxiliary task slots include:

• compression — the model that summarizes long context when the conversation approaches the context window limit
• vision — the model that handles image inputs and visual analysis (defaults to the primary model if not configured)
• session_search — the model that interprets natural language queries against the conversation archive
• approval — the model that evaluates pending skill and memory writes in the write_approval queue
• flush_memories — the model that compacts and consolidates MEMORY.md when it approaches its character limit

Budget models like DeepSeek V4 Flash ($0.14/million input tokens) or Gemini 3.1 Flash-Lite ($0.25/million input tokens) are well-suited to all of these side tasks. The quality difference between a premium model and a budget model for compression or session title generation is negligible. The cost difference is 10-20x.

Fallback Providers

Provider routing handles selection among available providers. Fallback providers handle the case where the primary model is unavailable — a rate limit, a provider outage, or an API error. You configure a fallback as a different model entirely, not just a different provider for the same model.

A common pattern: primary model is Claude Opus 4.8 via Anthropic. Fallback is Claude Sonnet 4 via OpenRouter. If Anthropic returns a rate-limit or timeout, Hermes automatically retries the request against the fallback without stopping the session. From the agent's perspective, the session continued normally. From your perspective, a transient API issue did not interrupt a long autonomous run.

Fallbacks are configured per model slot, not globally. Your primary model can have a different fallback than your auxiliary compression model. This granularity lets you design your fallback strategy for the relative importance and cost sensitivity of each task type.

Credential Pools

Rate limits are per API key. If your usage is high enough to hit rate limits on a single key, credential pools let you distribute API calls across multiple keys for the same provider. Hermes round-robins requests across the pool, distributing load evenly.

Credential pools are configured in config.yaml under the provider's credentials section. Each key in the pool is identified by a name and a value (typically loaded from ~/.hermes/.env rather than hardcoded). The pool is transparent to the rest of the configuration — you configure your model as you normally would, and Hermes handles the distribution.

This is primarily relevant for teams using a shared Hermes gateway, where multiple users' sessions share the same API credentials. For individual use, a single key is sufficient unless you are running intensive automated workloads.

Measuring Cost with /usage

All of these cost optimization techniques are only meaningful if you can measure their impact. The /usage slash command displays a breakdown of the current session: total input tokens, total output tokens, cache hit counts and savings, cost per model call, and cumulative session cost. The web dashboard extends this to historical periods — daily, weekly, and monthly breakdowns by model, by session, and by task type.

The right approach to Hermes cost optimization is empirical: measure your baseline, apply a change (add an auxiliary model, tune provider routing), measure again. Theoretical savings are useful for prioritization; actual savings require measurement. The /usage command and dashboard analytics give you the data to make that assessment.