Prompt caching cuts the cost of repeated tokens by roughly 75–90%, and batch APIs cut standard rates by around 50% for work that doesn’t need to be instant. They target different things — caching kills the cost of resending the same prefix, batching discounts the whole non-urgent job — so on the right workload the two discounts stack. The prompt caching savings calculator works out the combined number for your traffic.
What prompt caching does
Most real prompts have a large fixed part and a small changing part. A support bot might send 3,000 tokens of system instructions, tool definitions, and examples on every single request, then 100 tokens of the user’s actual question.
Without caching you pay full input price for all 3,100 tokens every time. Prompt caching lets the provider store that fixed prefix after the first call. On later calls, the cached portion is read at a steep discount — commonly 75–90% off the normal input rate — while only the fresh tokens pay full price.
Two things to know:
- There’s usually a small write cost. Creating the cache entry can cost slightly more than a normal read (often around 25% extra on that first write). You recover it within a few reuses.
- Caches expire. Entries live for a short window (minutes, sometimes longer). Caching pays off when the same prefix is hit repeatedly inside that window, not for one-off prompts.
The savings apply only to the repeated prefix. Keep the stable content at the front of the prompt and the variable content at the end so the cache boundary lands where it helps.
What batching does
A batch (or asynchronous) API takes a pile of requests and processes them off the real-time path. You give up instant responses and get results back within a window — often up to 24 hours. In exchange, providers commonly discount both input and output by around 50%.
Batching fits work with no user waiting on the other end: overnight document classification, bulk summarisation, evals, tagging a backlog, generating embeddings for a whole corpus. If a human is watching a spinner, batching is the wrong tool. If a queue is draining while everyone sleeps, it’s close to free money.
How they stack
The key point: caching and batching discount different axes, so they multiply rather than compete. Caching reduces the rate on repeated tokens; batching reduces the rate on the whole job. Run a cached prompt inside a batch job and both apply.
Worked example
A nightly job runs 10,000 requests. Each request shares a 2,000-token system prompt and adds 200 unique tokens. Assume an input rate of $3 per million tokens (illustrative — check live pricing). Ignore output for a moment to keep the prefix maths clear.
| Scenario | Effective input cost |
|---|---|
| No optimisation | 10,000 × 2,200 tok × $3/1M = $66.00 |
| Caching only (90% off the 2,000-token prefix) | prefix ≈ $0.60 + fresh 200 tok ≈ $6.00 = $6.60 |
| Batching only (50% off everything) | $33.00 |
| Caching + batching | ≈ $6.60 × 0.5 = $3.30 |
The cached prefix drops from $60 to about $6 because those 2,000 tokens are the same every time. Layering the ~50% batch discount on top roughly halves the remainder. Together they take a $66 job down to around $3.30 — a rough ~95% cut. Your real numbers will differ with rates, cache hit rate, and output volume, but the shape holds: caching handles the repetition, batching handles the urgency.
Which workloads actually benefit
- Long shared system prompt, many calls — the textbook caching win. Agents, chatbots, and RAG pipelines that resend the same instructions and context every turn.
- Non-urgent bulk jobs — the textbook batching win. Backfills, evals, overnight processing.
- Both at once — a batched job whose every request shares a big prefix gets the stacked discount.
Workloads that benefit least: one-off prompts with no repeated prefix (nothing to cache) and anything a user is actively waiting on (can’t batch).
If your cost is driven by long conversations rather than one big shared prefix, read why AI agents cost so much — caching the system prompt is one of the main levers there too. And once you’ve squeezed caching and batching, compare cheaper models on the same task with the model switch savings calculator. To size your own stacked discount first, start with the prompt caching savings calculator.
