Timeouts, Stale Detection, and Production Hardening

Why Timeouts Matter in Production

In a batch job processing 1,000 prompts, a single hanging request can block a worker process indefinitely. If that worker is waiting for a response that will never come -- because the provider is down, the network connection dropped, or the model got stuck in an infinite loop -- that worker sits idle while the other workers continue processing. Eventually, if enough workers hang, your batch throughput grinds to a halt.

Timeouts are the defense against this. By setting a maximum time to wait for a response, you ensure that stuck requests fail fast, release their worker, and get handled by fallback logic or retried. A batch job with proper timeouts configured can handle provider failures gracefully. One without timeouts can hang for hours.

Request Timeout vs. Stale Timeout

Hermes uses two different timeout mechanisms, each catching a different failure mode:

Request timeout is the maximum total time for an API call to complete -- from the moment Hermes sends the request until it receives the complete response. If this timeout fires, the entire request is cancelled. The default is 1,800 seconds (30 minutes) for most providers, which is generous enough to handle very long inference runs. Local providers automatically get a higher limit since they are often slower.

Stale timeout is a streaming-specific mechanism. When Hermes is receiving a streaming response, it monitors the gap between token deliveries. If no tokens arrive for longer than the stale timeout, Hermes assumes the connection has frozen and cancels the request. The default stale timeout is 300 seconds (5 minutes) for non-local providers. Local providers have stale detection disabled by default.

The key difference: the request timeout bounds the total duration; the stale timeout catches frozen connections that are still technically open but not producing output. Both are needed because a frozen streaming connection can remain open indefinitely without triggering the request timeout.

Configuring Timeouts

Timeout configuration in config.yaml operates at three levels: provider-wide defaults, model-specific overrides, and per-request overrides via CLI.

Provider-wide:

providers:
  openrouter:
    request_timeout_seconds: 300
    stale_timeout_seconds: 120

Model-specific (overrides the provider-wide setting for that model):

providers:
  openrouter:
    request_timeout_seconds: 300
    models:
      o3:
        timeout_seconds: 1800

The model-specific timeout_seconds overrides the provider's request_timeout_seconds for that specific model. This is useful for models that are expected to be slow (reasoning-heavy models, large context models) while keeping a tight timeout for everything else.

Choosing the Right Timeout Values

Setting timeouts too low causes valid requests to fail. Setting them too high lets stuck requests block workers for too long. The right values depend on your workload:

For batch processing, err on the side of tighter timeouts. A failed request that gets retried is better than a hanging request that blocks a worker for 30 minutes. With a fallback chain in place, a timeout triggers a retry on the backup provider, and the total cost is a small delay rather than a stuck worker.

Socket Read Timeout

Below the API-level timeouts, there is a socket-level read timeout that limits how long Hermes waits for data on the underlying network connection. The default is 120 seconds. This timeout catches cases where the network connection is technically open but no data is flowing -- a different failure mode from stale detection (which monitors token delivery at the API level).

The socket read timeout is automatically raised to 1,800 seconds for local providers, matching the expectation that local inference might have slow data transfer rates. You typically do not need to configure this directly; it is part of the built-in defaults that Hermes manages appropriately based on the provider type.

A Production Configuration Checklist

Before running a large batch job or going live with a Hermes deployment, run through this checklist:

Credentials. API keys are in .env, not in config.yaml. Test them with a single interactive session before running the batch. Confirm the docker_forward_env list includes every key needed inside containers.

Fallback chain. At least one fallback provider is configured. You have tested it by temporarily breaking the primary. The fallback uses a different provider from the primary (not just a different key on the same provider).

Timeouts. Request timeout and stale timeout are configured for the expected request duration. Model-specific overrides are in place for any slow reasoning models. Local providers are not being given the same timeout as cloud providers.

Credential pools. If multiple keys are configured, the pool strategy is explicitly set. You have confirmed the keys represent independent rate limit budgets if rate limit avoidance is the goal.

Auxiliary routing. Auxiliary tasks are routed to appropriate models. Timeout values on auxiliary tasks are shorter than on the main model. Fallback chains exist for auxiliary tasks that must not fail.

Prompt caching. You are using a supported provider (Anthropic, OpenRouter, Nous Portal). The cache TTL is set to "1h" unless you have a specific reason for "5m".

Test run. You have run a 20-50 prompt subset of your batch dataset and inspected the results. No systematic errors were observed. Trajectory quality looks acceptable.

A batch job that clears this checklist has a high probability of completing successfully. One that skips these steps has a meaningful risk of hanging, producing poor-quality results, or failing silently partway through.