Reliability and Cost Control: Provider Routing, Fallback Providers, and Credential Pools

Why This Matters More Once an Agent Runs Unsupervised

When you are watching a CLI session yourself, a rate limit error is a minor annoyance -- you see it, you wait, you retry. A cron job running at 3am or a gateway answering a message while you are asleep does not have you there to notice and retry manually. This lesson covers the three layered systems that keep Hermes working through exactly those situations without you intervening.

The Three Layers, in Order

Per Hermes' own documentation, resilience works in three layers: credential pools rotate between multiple keys for the same provider first, primary model fallback switches to an entirely different provider if that does not resolve things, and auxiliary task fallback handles side tasks (vision, compression, and similar) with independent provider resolution of their own.

Layer 1: Credential Pools

Register multiple API keys or OAuth tokens for the same provider, and Hermes rotates between them as needed rather than failing the moment one hits a limit. This is the one place in this lesson where you genuinely need to go get something new yourself: a second (or third) key from the same provider you are already using -- generated the same way you got your first key, from that provider's own dashboard, often under a second account if the provider does not allow multiple keys per account. This is optional; one key per provider is enough to use Hermes normally, credential pools are specifically for higher-volume or unattended use where hitting one key's limit would otherwise stall everything.

The actual flow: pick a key using your chosen strategy, send the request, then react based on what comes back.

• 429 (rate limited) -- retries the same key once, in case it was transient; if it fails again, rotates to the next key with a 1-hour cooldown on the failed one
• 402 (billing/quota) -- rotates immediately, with a 24-hour cooldown
• 401 (auth failure) -- attempts an OAuth token refresh first; only rotates if that refresh itself fails
• Pool exhausted -- falls through to your configured fallback provider (Layer 2)

A successful call resets the retry-tracking flag, so one transient blip does not permanently mark a key as suspect.

Rotation strategy is configurable:

credential_pool_strategies:
  openrouter: round_robin
  anthropic: least_used

Options: fill_first (default -- use the first healthy key until it is exhausted), round_robin (cycle evenly), least_used (always pick the lowest-usage key), random.

Adding keys:

hermes auth add openrouter --api-key sk-or-v1-your-key
hermes auth list

Layer 2: Fallback Providers

If credential pool rotation is not enough -- or you only have one key per provider in the first place -- fallback providers switch to a different provider and model entirely. This triggers on rate limits and server errors (429, 500, 502, 503) after retries are exhausted, on auth failures (401, 403) immediately, and on malformed API responses.

The mechanism swaps model, provider, and client in-place while preserving your conversation history -- nothing about the actual conversation is lost in the switch. One important scope limit: fallback is per-turn only. It activates at most once per message, and the next message you send starts fresh with your primary model again rather than staying on the fallback indefinitely.

Set up interactively:

hermes fallback        # add / list / remove / clear

Or manually in config.yaml:

model:
  provider: anthropic
  default: claude-sonnet-4-6
fallback_providers:
  - provider: openrouter
    model: anthropic/claude-sonnet-4

You can even fall back to a fully local model:

fallback_providers:
  - provider: custom
    model: llama-3.1-70b
    base_url: http://localhost:8000/v1
    key_env: LOCAL_API_KEY

Layer 3: Auxiliary Task Fallback

Side tasks -- vision, web extraction, compression, the skills hub, MCP, the approval system, title generation -- each resolve their own provider independently, rather than all being tied to whatever your main model is. When a task is set to "auto", the resolution chain tries: the main provider, then a task-specific fallback chain if one is configured, then your top-level fallback_providers list, then built-in discovery as a last resort.

Provider Routing: Controlling Cost and Speed Deliberately

Separate from failure handling, provider routing lets you actively choose how OpenRouter selects between providers for a given model -- this only applies when you are connecting through OpenRouter, not direct provider APIs. Configure it under provider_routing in config.yaml:

• sort -- rank by "price", "throughput", or "latency"
• only -- whitelist specific providers (e.g. restrict to Anthropic and Google only)
• ignore -- blacklist specific providers
• order -- explicit priority ranking
• require_parameters -- only route to providers supporting every parameter in your request
• data_collection -- "allow" or "deny", controlling whether a provider can use your prompts for training

Real combinations: sort: "price" for high-volume usage where cost matters most; sort: "latency" for interactive applications where time-to-first-token matters most; only: ["Anthropic"] if you specifically need provider lock-in for compliance reasons; combining ignore with data_collection: "deny" when privacy matters and you want both certain providers excluded and a training-data opt-out enforced. These transmit to OpenRouter via the extra_body.provider field on every API call, and the configuration loads once at startup, whether you are running the CLI or the gateway.

The next lesson covers extending Hermes with your own custom code: event hooks for reacting to lifecycle events, and the full plugin system for adding entirely new tools and commands.