Custom Endpoints and Self-Hosted Models

Why Use Local or Custom Endpoints

Not every workload belongs on a cloud API. There are several scenarios where you want to run inference against a local model or a custom endpoint rather than a provider like Anthropic or OpenRouter:

Privacy. Some data cannot leave your infrastructure. Medical records, legal documents, proprietary code -- if your use case involves sensitive data, running a local model ensures nothing is sent to a third party.

Cost at scale. At high enough volume, self-hosting a model can be cheaper than cloud API costs. The economics vary by model size, hardware, and usage patterns, but it is a real consideration for heavy batch workloads.

Offline operation. A local model works without an internet connection. For edge deployments, air-gapped environments, or situations where reliability matters more than model quality, local inference is the only option.

Custom models. If you have fine-tuned a model on your own data (perhaps using trajectories from Hermes's batch processing), you want to run inference against your fine-tuned weights, not a generic cloud model.

The Universal Custom Endpoint Pattern

Hermes connects to any OpenAI-compatible API endpoint using the base_url configuration. When you set base_url, the provider field is ignored -- Hermes sends requests directly to your endpoint using OpenAI-compatible format:

model:
  provider: custom
  base_url: "http://localhost:1234/v1"
  api_key: "local-key"

The api_key field is passed as the Authorization header. For many local inference servers, any non-empty string works -- they do not validate the key. For others, you configure a specific key in the server's settings. Use whatever your server expects.

Popular local inference servers that expose OpenAI-compatible APIs include Ollama (default port 11434), LM Studio (default port 1234), and vLLM (configurable, commonly 8000). All three work with this configuration pattern.

Specifying the Model for Local Servers

For local inference, the model field should match how your local server identifies the model. This varies by server:

• Ollama uses the model tag you pulled: model: "llama3.2:3b" or model: "qwen2.5-coder:7b"
• LM Studio uses the model's identifier as shown in the LM Studio UI
• vLLM uses the Hugging Face model ID: model: "meta-llama/Llama-3.2-3B-Instruct"

If Hermes is sending requests to your local server but getting errors, the model name mismatch is the most common cause. Check your server's logs to see what model names it expects.

Custom Endpoints for Auxiliary Tasks

The base_url override works for auxiliary tasks as well as the main chat model. This is how you route side tasks to a local model while keeping the main conversation on a cloud provider:

model: anthropic/claude-opus-4
auxiliary:
  compression:
    base_url: http://localhost:11434/v1
    model: llama3.2:3b
    api_key: ollama

Context compression goes to Llama 3.2 3B running locally (fast and free), while main reasoning uses Claude Opus 4 (expensive but necessary for quality). This is a cost-effective pattern for workflows where the side tasks do not require the main model's capability.

Stale Detection and Local Providers

Hermes includes a stale detection mechanism that monitors streaming responses. If a streaming response stops producing tokens for longer than the stale timeout (default 300 seconds for most providers), Hermes assumes the connection has stalled and cancels the request.

Local models are often much slower than cloud APIs, particularly during the prefill phase when the model is processing a long prompt. A local model might take 60 to 120 seconds just to start generating the first token. If stale detection fires during that prefill phase, it cancels a request that was actually making progress -- just slowly.

Hermes handles this automatically: when you configure a local endpoint (one with a localhost or private network base_url), stale detection is automatically disabled. You do not need to set a timeout override for local providers; the auto-detection takes care of it.

For custom endpoints that are remote but slow (an underpowered cloud VM, a high-latency self-hosted deployment), you may want to explicitly raise the stale timeout:

providers:
  custom:
    stale_timeout_seconds: 600

Setting it to a high value prevents false-positive cancellations on slow but functional endpoints.

Docker and Credential Forwarding for Batch Runs

When batch processing uses Docker containers for isolation (via the docker_image field in your dataset), environment variables do not automatically flow into the container. If your prompts need to call a local endpoint, and the endpoint's URL or credentials need to be available inside the container, you must explicitly forward them.

The docker_forward_env list in config.yaml specifies which environment variables to inject:

docker_forward_env:
  - OPENROUTER_API_KEY
  - LOCAL_MODEL_KEY
  - CUSTOM_TOOL_ENDPOINT

Only the listed variables are forwarded. This is intentional: explicit forwarding is a security feature that prevents accidentally exposing credentials to containers that do not need them.

For local model endpoints, you typically need to forward the endpoint URL if it is stored as an environment variable, and any authentication key the local server requires. If your local server requires no authentication, just forward the URL.

OAuth Providers

Some providers use browser-based OAuth authentication instead of API keys. Hermes supports OAuth for several providers -- ChatGPT (OpenAI Codex), MiniMax, and xAI Grok OAuth -- via the hermes model command, which opens a browser window for the OAuth flow. Tokens are stored in ~/.hermes/auth.json and automatically refreshed.

OAuth providers do not use the custom endpoint pattern -- they use their own provider identifiers (openai-codex, minimax-oauth, xai-oauth) and manage authentication through the stored token. You configure them the same way as any other provider, just without an API key in .env.