Prompt Caching: Cross-Session Memory That Saves Money

The Problem: Paying for the Same Tokens Every Session

Every Hermes session starts by sending the full system prompt to the model. That system prompt includes your agent's instructions, the active skills, tool definitions, and any context files you have configured. Depending on your setup, this can easily be 5,000 to 15,000 tokens or more -- and you pay for those tokens on every single API call, including the first call of every new session.

Prompt caching addresses this directly. Instead of re-sending the full system prompt on every call, the provider caches the prompt's content on their servers. Subsequent calls that use the same prompt prefix pay a significantly lower token rate -- or in some cases, nothing for the cached portion. The cache persists across sessions, so you pay the full price once and get the discount for the lifetime of the cache.

This is not a Hermes-specific concept -- it is a capability offered by supported AI providers. But Hermes implements it automatically, with no configuration required to get the basic benefit.

What Hermes Caches

Hermes applies cache control breakpoints to two parts of every session's prompt:

The system prompt. Your agent's core instructions, personality configuration, and any static context. This is the largest and most stable part of the prompt -- it changes rarely between sessions, making it an ideal caching target.

Skill blocks. The content of active skills loaded into the session. Each skill adds content to the context that tells the agent how to use that skill. Since skills change only when you explicitly add, remove, or update them, skill content is highly cacheable.

These two components together typically represent 80% or more of the total token cost of the first API call in a session. Caching them means every follow-up call in the same session -- and every future session that uses the same system prompt and skills -- pays substantially less.

Automatic Activation: Nothing to Configure

On supported providers, Hermes enables prompt caching automatically. You do not need to add any configuration to start benefiting from it. The cache control breakpoints are inserted into the API calls automatically when the provider supports them.

This automatic behavior is the right default. Prompt caching is unambiguously beneficial on supported providers -- it reduces costs with no downside. Making it opt-in would mean most users would miss the savings without realizing it.

The automatic cache activates for three providers: native Anthropic API, OpenRouter, and Nous Portal. If you are using any of these, caching is already active. You can confirm by checking your provider's usage dashboard for cache hit metrics.

Cache TTL: The One Setting You Can Configure

The default cache TTL (time-to-live) is 1 hour on supported providers. This means cached prompt content is stored on the provider's servers for one hour. If a new session starts within that hour and uses the same system prompt and skills, it gets the cached discount. If an hour passes without a request, the cache expires and the next session pays full price again.

Anthropic supports two TTL tiers: 5 minutes and 1 hour. You can configure which tier Hermes uses:

prompt_caching:
  cache_ttl: "5m"

or

prompt_caching:
  cache_ttl: "1h"

The 5-minute TTL costs less to store but expires quickly -- only useful if you are running rapid successive sessions within a 5-minute window. The 1-hour TTL is the right choice for most users: it costs slightly more to cache but provides the discount for much longer, which is better for typical usage patterns where sessions are spread over the course of an hour or a day.

The default of "1h" is correct for the vast majority of use cases. You would only lower it to "5m" if you are running very high-frequency batch processing and want to minimize cache storage costs.

Provider-Specific Behavior

Each supported provider implements prompt caching slightly differently:

Native Anthropic API supports the full cache_ttl configuration with both 5-minute and 1-hour tiers. Cache hits are reported in the API response so you can confirm caching is working. The pricing discount for cached tokens is approximately 90% off the standard input token price.

OpenRouter supports prompt caching and passes it through to underlying providers that support it. The behavior depends on which Anthropic or other model you are routed to. OpenRouter reports cache statistics in its usage data.

Nous Portal supports prompt caching automatically. Configuration follows the same pattern as native Anthropic.

Qwen Cloud supports caching but caps the TTL at 5 minutes automatically. Even if you configure cache_ttl: "1h", Qwen Cloud enforces its own 5-minute maximum. Hermes handles this transparently -- you do not need a separate configuration for Qwen, but do not expect 1-hour cache lifetimes when using it.

xAI Grok uses a different mechanism entirely: session-pinned conversation IDs rather than explicit cache control. The effect is similar -- repeated calls within a session share state -- but the implementation is different. The cache_ttl setting does not apply to xAI Grok.

When Caching Saves the Most

The savings from prompt caching are proportional to how often the same cached prefix is reused. A few patterns produce the most benefit:

Long sessions with many turns. A 30-turn conversation in a single session reuses the system prompt and skill cache on turns 2 through 30. The first turn pays full price; the remaining 29 pay the cached rate. With a large system prompt, this produces substantial savings.

Rapid successive sessions. If you run multiple interactive sessions within an hour (or if a batch job starts many sessions in quick succession), the cache TTL means most of those sessions start from a cache hit rather than full-price tokens.

Large skill libraries. Users with many active skills have larger skill blocks in their system context. A large skill library that is stable (you rarely change which skills are active) is a good cache target -- you pay full price once, then get the discount on every subsequent session until the skills change.

Checking Your Cache Performance

Most providers expose cache statistics in their API responses or dashboards. After running some sessions, check your provider's usage dashboard to see the cache hit rate. A high hit rate (above 80%) means caching is working well. A low hit rate suggests your system prompt or skills are changing frequently between sessions, which prevents cache reuse.

If you are seeing low hit rates unexpectedly, check whether you have configurations that vary the system prompt dynamically between sessions -- personalization settings, dynamic context injection, or profiles with different skills active. Each unique system prompt variant needs its own cache entry, which reduces the benefit of caching if you have many variants.