Credential Pools and Key Rotation

The Problem Credential Pools Solve

API providers enforce rate limits. When you hit a rate limit, your requests get rejected with a 429 error until the limit resets -- usually after a minute or an hour. For interactive sessions, this is annoying but manageable. For a batch run processing 1,000 prompts in parallel, a rate limit can halt your entire job for an extended period.

One common solution is to get more API keys. If you have four keys, you have (roughly) four times the rate limit capacity. But managing four keys manually -- deciding which to use for which request -- is tedious and error-prone. You might overuse one key while the others sit idle, or consistently direct expensive requests to your most expensive key.

Credential pools automate this. You provide multiple keys for a provider, configure a rotation strategy, and Hermes handles the distribution automatically. Each request is routed to a key according to the strategy, and you get the aggregate capacity of all your keys without any manual management.

Configuring a Credential Pool

Credential pool configuration lives in config.yaml. You first need to store multiple keys for the same provider in your .env file, then reference them in the pool configuration. The convention for multiple keys for the same provider is a numbered suffix:

OPENROUTER_API_KEY=sk-or-v1-primary
OPENROUTER_API_KEY_2=sk-or-v1-secondary
OPENROUTER_API_KEY_3=sk-or-v1-tertiary

Then in config.yaml:

credential_pool_strategies:
  openrouter: round_robin

Hermes automatically discovers all keys matching the PROVIDER_API_KEY naming pattern and adds them to the pool. The strategy tells it how to distribute requests across those keys.

The Four Rotation Strategies

Hermes supports four strategies for rotating between keys in a pool. Each has a different use case:

fill_first (the default) always uses the first key until it hits a rate limit, then falls through to the next. This concentrates usage on one key, which is useful when you want to maximize free tier usage before dipping into a paid key. The first key burns its rate limit, the second becomes active, and so on down the list.

round_robin cycles through keys in sequence: request 1 uses key 1, request 2 uses key 2, request 3 uses key 3, then back to key 1. This distributes requests evenly across all keys in the pool. For batch processing where you want to use all keys equally, this is usually the right choice.

least_used always routes the next request to the key that has handled the fewest requests so far. This is similar to round_robin in steady-state but self-corrects: if one key is slower (due to rate limiting or latency), it receives fewer requests automatically, and the others pick up the slack. For long-running batch jobs, this often produces better balance than round_robin.

random selects a key randomly for each request. The distribution converges to even over many requests, but there is no guarantee of balance for small batches. Useful for environments where you want to avoid any predictable pattern in key usage, but round_robin or least_used is usually preferable for performance.

Choosing a Strategy

The right strategy depends on what you are trying to accomplish:

For most batch processing workloads, least_used is the recommended default. It adapts to real-world conditions (some keys slower than others, some hitting rate limits) without requiring any manual adjustment.

Per-Provider Configuration

You can configure different strategies for different providers in the same config block:

credential_pool_strategies:
  openrouter: round_robin
  anthropic: least_used
  nous: fill_first

This is useful when you use different providers for different purposes. Your OpenRouter pool might serve batch processing where even distribution matters most. Your Anthropic pool might serve your main interactive sessions where least_used adaptive balancing is better. Your Nous pool might have one primary and one backup key where fill_first makes the intent clear.

Key Pools for Team Environments

Credential pools are especially valuable in team settings where multiple developers share a Hermes deployment. Instead of each developer managing their own API key, a shared configuration uses a pool of keys from a shared account. The pool distributes requests across keys, preventing any single developer's usage pattern from exhausting the team's rate limits.

The team configuration is stored in a shared config.yaml that specifies the pool strategy. Each developer's .env file contains the actual keys. Or, if the deployment is a shared server, a single .env file holds all keys and all team members use the same pool automatically.

In this setup, the keys themselves are the sensitive information that should not be shared broadly. The pool strategy configuration in config.yaml is safe to share -- it describes how keys are used but does not contain the keys themselves.

What Credential Pools Do Not Do

Credential pools manage distribution but do not guarantee rate limit avoidance. If all four keys in your pool belong to the same account, they share rate limits. Adding more keys from the same account does not increase your aggregate rate limit -- it just changes how requests are routed to keys that are already sharing a cap.

True rate limit capacity increase requires keys from different accounts or upgrading to a higher-tier plan. Credential pools are most effective when the keys genuinely represent independent rate limit budgets -- different accounts, different organizations, or different API products with their own caps.

Also note that credential pools do not provide automatic failover if a key becomes invalid. If one key in the pool gets revoked or expires, requests routed to that key will fail. Monitor your keys' validity regularly and remove expired keys from your pool configuration promptly.