LLM token cost optimization is not simply a matter of choosing the cheapest model. This guide shows how to estimate recurring inference costs, trim prompts and retrieved context, use caching and batching appropriately, and route requests by difficulty without sacrificing response quality. The method is designed for developers reviewing an existing workflow or planning an LLM app for production.
Overview
Most LLM API bills are driven by four variables: input tokens, output tokens, request volume, and the price attached to each token category. Some providers also distinguish cached input, batch processing, model tiers, or other usage modes. Because these inputs change over time, a useful cost model should store them as editable assumptions rather than hard-code them into application logic.
Cost reduction should be treated as an engineering trade-off. A shorter prompt may reduce quality if it removes necessary instructions or evidence. A smaller model may be adequate for classification but unsuitable for complex reasoning. Aggressive output limits can lower spend while causing truncation or retries. The goal is therefore not the fewest tokens in isolation; it is the lowest cost per acceptable result.
Start by defining what “acceptable” means for the workflow. For a support classifier, it might mean a minimum agreement rate with reviewed labels. For a retrieval-augmented generation (RAG) assistant, it may include answer correctness, citation coverage, and refusal behavior. Use a repeatable evaluation set before applying cost changes. The LLM prompt testing evaluation framework can help organize those checks.
How to estimate
Use a simple per-request model first:
Cost per request = (input tokens × input rate) + (output tokens × output rate)
Then estimate the period total:
Period cost = cost per request × requests in the period
If your provider uses different rates for cached input, batch jobs, or model variants, calculate each category separately. For example:
Total cost = standard input cost + cached input cost + output cost + batch or platform charges
Do not mix token units accidentally. A provider may quote a rate per million tokens while your logs report tokens per request. Convert all rates to the same unit before multiplying. A spreadsheet or small internal calculator should include fields for average input tokens, average output tokens, requests per day, active days, retry rate, and the current rate for each usage category.
Retries deserve explicit treatment. If 8% of requests are retried, a first approximation is:
Adjusted requests = successful requests × (1 + retry rate)
This is only an estimate when retries have different prompt sizes or use a different model. For better accuracy, measure first attempts and retries as separate request types.
Inputs and assumptions
Capture the following inputs from production logs or a representative test run:
- Request volume: Separate interactive traffic, scheduled jobs, background enrichment, and development tests.
- Input tokens: Measure system instructions, user content, conversation history, tool schemas, and retrieved documents separately where possible.
- Output tokens: Record both the average and a high percentile or maximum budget. Long-tail outputs can drive unexpected costs.
- Model mix: Identify which requests use each model and why.
- Retries and failures: Include timeouts, invalid structured output, tool-call failures, and application-level retries.
- Quality constraints: Record the evaluation metric that a cheaper configuration must continue to meet.
- Pricing inputs: Enter rates from the provider’s current documentation at the time of review. Avoid copying rates into a permanent article or code comment without a date.
Prompt optimization for cost usually begins with context trimming. Remove duplicated instructions, stale conversation turns, verbose tool descriptions, and retrieved passages that do not affect the answer. Preserve constraints, definitions, output schemas, and evidence needed for a correct result. If structured output is required, keep the schema precise and test it after every reduction; see Structured Output Prompting for practical guidance.
For RAG workflows, retrieve fewer but more relevant chunks instead of sending an entire document collection. Apply metadata filters, deduplicate overlapping passages, and consider a second-stage reranker when it reduces irrelevant context reliably. A similarity tool or embedding evaluation can help compare retrieval changes, as discussed in the text similarity checker guide.
| Technique | What it changes | Best fit | Risk to monitor |
|---|---|---|---|
| Prompt trimming | Removes redundant instructions and examples | Repeated production prompts | Lost constraints or inconsistent behavior |
| Context filtering | Sends fewer retrieved or historical tokens | RAG and long conversations | Missing evidence or reduced answer coverage |
| Output limits | Caps generated tokens | Classifiers, summaries, structured responses | Truncation and retries |
| Caching | Reuses stable prompt or computation components where supported | Shared system instructions and repeated prefixes | Stale content or cache-ineligible requests |
| Batching | Groups non-urgent work into a supported batch mode | Nightly enrichment and back-office jobs | Higher latency or operational complexity |
| Model routing | Assigns simple tasks to less capable models | Mixed workloads | Quality failures on edge cases |
Worked examples
Assume a workflow processes 10,000 requests in a review period. Its average request contains 2,400 input tokens and 350 output tokens. Do not insert a provider price until you have verified the current rate card. Instead, define symbolic rates: Ri for input cost per token and Ro for output cost per token.
The baseline estimate is:
10,000 × (2,400Ri + 350Ro)
Suppose testing reduces retrieved context by 25% and output length by 10%, while request volume and quality remain acceptable. The revised estimate becomes:
10,000 × (1,800Ri + 315Ro)
The percentage reduction can then be calculated from the two estimates:
Saving percentage = (baseline cost − revised cost) ÷ baseline cost × 100
This approach is more useful than claiming a universal saving because the result depends on the provider’s input-to-output price ratio. If output tokens are much more expensive than input tokens, output controls may have greater effect. If input dominates because of large RAG context, retrieval filtering may matter more.
For model routing, divide the workload into categories such as extraction, classification, summarization, and complex generation. Estimate each category separately, then test a routing rule against the same evaluation set. A practical rule might send deterministic extraction to a lower-cost model, use a general model for ordinary requests, and escalate only when confidence is low or required fields are missing. Log the route, token counts, latency, retries, and quality result so that the rule can be reviewed rather than treated as permanent.
Caching and batching should be evaluated operationally as well as financially. Caching helps most when a large part of the request is stable and the provider supports the relevant cache behavior. Batching is better suited to work that can tolerate delay. Neither should be assumed to reduce total cost until measured under the provider’s current terms and your actual request pattern.
When to recalculate
Revisit the estimate whenever a pricing input, model limit, or workload pattern changes. In practice, recalculate before a budget review, after changing models, when a context window or caching feature changes, and after launching a feature that increases conversation history, retrieval volume, or tool usage.
Set a recurring review using these checks:
- Export recent usage by model, route, request type, and retry status.
- Compare actual input and output token distributions with the assumptions in the cost model.
- Verify current provider pricing, supported caching modes, batch terms, and context limits.
- Run the quality evaluation set against any trimmed prompt, new model, or routing rule.
- Calculate cost per successful acceptable result, not only cost per API call.
- Record the decision, assumptions, date, and next review trigger.
Finally, add observability before optimizing aggressively. The LLM app production checklist covers cost, security, and monitoring considerations that belong alongside token accounting. A transparent model of usage makes cost reductions easier to verify, easier to reverse, and safer to apply as APIs and workloads evolve.
