Prompt Caching and Cost Optimization

The Caching Price Breakdown

Every request to the Astra API involves at least two costs: input tokens and output tokens. Prompt caching adds two more line items — cache writes and cached reads — but those additions exist to make the total cost significantly lower when the same content appears across multiple requests.

The four rates you need to know:

• Fresh input: $10 per million tokens. Every token in a request that isn't cached bills at this rate.
• Cache write: $12.50 per million tokens. The first time a prompt prefix is cached, you pay a 25% premium over the fresh rate. This is the one-time cost of establishing the cache entry.
• Cached read: $1 per million tokens. Every subsequent request that hits that cache entry bills the cached portion at $1 instead of $10 — a 90% reduction.
• Output: $50 per million tokens. This rate applies to everything the model generates and is unaffected by caching.

A concrete example shows why this matters. Suppose you have a 10,000-token system prompt — a detailed persona, tool definitions, background context — sent on every request. Across 1,000 requests:

• Uncached: 1,000 requests × 10,000 tokens = 10,000,000 input tokens × $10/M = $100.
• With caching: One cache write (10,000 tokens × $12.50/M = $0.13) plus 999 cached reads (9,990,000 tokens × $1/M = $9.99). Total: $10.12.

The cache write is a one-time overhead. Every read after it costs one-tenth of the uncached rate. At 1,000 requests the caching saves roughly $90 on that prompt alone — and the savings compound as volume grows. OpenAI applies caching automatically for any prompt prefix of 1,024 tokens or longer; no special API parameter is required.

Designing for Cache Hits

The cache key is the exact prefix up to the first token that changes between requests. Anything that varies breaks the cache at that point. The rule is: put everything that doesn't change at the top, before anything that varies. In practice that means ordering your prompt from most stable to least stable:

1. System prompt first. Your persona, constraints, and task instructions stay constant. They go at the very top of the input.
2. Tool definitions second. Tool schemas rarely change between requests for the same application. Include them immediately after the system prompt.
3. Background context third. Documents, knowledge-base excerpts, or reference material that applies to every request in a session belong here.
4. Conversation history fourth. Prior turns are relatively stable within a session; they go after the static content.
5. Current user message last. This changes on every request, so it goes at the end. Everything above it is eligible for caching.

Two design mistakes break cache hits unnecessarily. The first is injecting a dynamic value — a timestamp, a session ID, a request counter — into the system prompt. A single varying byte near the top of the prompt invalidates everything that follows. If you need that value in the prompt, move it to a separate user-turn message after all the static content. The second mistake is regenerating the system prompt from a template on every request when the output is identical: string interpolation can introduce invisible whitespace changes or encoding differences that break the exact-match requirement. Assemble the static prefix once, store it as a string, and reuse that same object across requests. Treat it as immutable once it's working.

Reasoning Effort Cost Math

Output tokens cost $50 per million, which makes the cost-per-task calculation sensitive to how many times the model runs before you accept the result. A higher reasoning tier often produces fewer retries because the model catches errors internally — but whether that tradeoff is worth it depends on your actual retry rate, not on intuition.

Consider a task that generates roughly 500 output tokens per attempt at medium effort, with three attempts needed before the result is acceptable. At max effort the same task completes in one attempt with 200 output tokens, because the additional reasoning iterations resolve the issue internally:

In this scenario max effort costs 4.5× less per accepted task than medium. The crossover point depends on your actual retry rate — a task where medium succeeds 95% of the time on the first attempt will almost certainly be cheaper at medium. The only way to know is to measure.

A few lines of instrumentation give you the numbers you need to make this decision from data rather than by feel:

def log_task_cost(response, tier, attempt):
    u = response.usage
    input_cost  = u.input_tokens  * 10 / 1_000_000
    output_cost = u.output_tokens * 50 / 1_000_000
    cached_save = getattr(u, "cached_tokens", 0) * 9 / 1_000_000
    print(f"tier={tier} attempt={attempt} "
          f"in=${input_cost:.5f} out=${output_cost:.5f} "
          f"cache_saved=${cached_save:.5f}")

Run this for a week on a frequently-used task class. If you see repeated retries pushing medium above what max would have cost, switch that task class to max and re-measure. If medium is consistently first-pass on a task you've been running at high, drop it down and recapture the margin.

Batch API: Half Price for Non-Urgent Work

The Batch API runs Astra at half the standard price: $5 per million input tokens and $25 per million output tokens. The tradeoff is throughput: batch jobs are processed asynchronously with a turnaround of up to 24 hours. If that delay is acceptable, the Batch API is the single highest-leverage cost lever available — a 50% reduction that requires no changes to prompt design, reasoning tier selection, or application logic.

Batch requests are submitted as a JSONL file where each line is an independent request object with its own input and model parameters. OpenAI processes the file and returns a results JSONL when complete. The pattern fits any workload where you process a queue of items rather than responding in real time:

• Classification at scale — categorizing support tickets, tagging documents, labeling datasets.
• Batch extraction — pulling structured fields from a large set of records or documents.
• Scheduled report generation — nightly or weekly summaries produced on a cron schedule rather than on demand.
• Evaluation pipelines — LLM-as-judge scoring runs that measure quality across a test set.

For these workloads, the decision is simply whether the pipeline can tolerate a 24-hour wait. If it can, there's no justification for paying the real-time rate. Batch also stacks with caching: static prefixes in batch requests still hit the cache at $1/M once established, so the two strategies compound on the input side.

When to Route to a Cheaper Model

Astra is easiest to justify when a task requires capabilities that only Astra has: computer use, a 1M-token context window, SRE-level agentic coding, or problems where advanced reasoning meaningfully reduces the retry rate. For tasks outside that set, testing a cheaper model first is a straightforward cost decision — GPT-5.6 costs roughly one-third of Astra at standard rates.

A practical routing decision tree:

1. Does the task require computer use? Astra is required.
2. Does the task require more than 128K tokens of context? Astra is required — GPT-5.6 tops at 128K.
3. Does the task involve multi-step agentic coding or SRE-level autonomy? Astra at high or xhigh is likely worth it.
4. Is the task classification, extraction, simple drafting, or a predictable transformation? Test GPT-5.6 first. If quality is acceptable, move that task class permanently.

Implement routing as a lookup rather than hardcoding the model in every call:

def route_model(task_type: str) -> tuple[str, str]:
    """Return (model_id, reasoning_effort) for a given task type."""
    routing = {
        "computer_use":   ("gpt-6-astra",  "high"),
        "long_context":   ("gpt-6-astra",  "medium"),
        "agentic_coding": ("gpt-6-astra",  "high"),
        "classification": ("gpt-5.6-sol",  "low"),
        "extraction":     ("gpt-5.6-sol",  "low"),
        "simple_draft":   ("gpt-5.6-sol",  "medium"),
    }
    return routing.get(task_type, ("gpt-6-astra", "medium"))

Keep the routing table in a config file rather than inline — the right model for a task class can change as pricing or quality shifts, and you want to update it in one place. Log which route was taken per task alongside your cost data. A route where quality complaints are clustering is a signal that the cheaper model isn't meeting the bar for that class; a route where Astra is overkill is a margin recovery waiting to happen.