Moving an LLM application from a working prototype to production requires more than selecting a model and adding an API key. This checklist gives development teams a repeatable way to review security, reliability, observability, privacy, and operating cost before launch, then recalculate those decisions as usage, prompts, and provider pricing change.
Overview
A production LLM application is a software system with uncertain inputs, variable outputs, external dependencies, and usage-based costs. The launch decision should therefore combine two reviews: a readiness checklist and a simple operating model.
The readiness checklist asks whether the application can protect data, constrain behavior, recover from dependency failures, and give operators enough information to investigate problems. The operating model estimates how much each request costs and how that cost changes with traffic, prompt length, output length, retries, and model selection.
Use this guide for chat interfaces, document-processing services, retrieval-augmented generation (RAG) applications, internal assistants, and automated workflows. It is deliberately provider-neutral because model capabilities, API pricing, limits, and product terms can change. Confirm the current details in the documentation for every model and service in your stack.
For prompt quality, pair this deployment review with a repeatable LLM prompt testing framework. A prompt that performs well in a demo still needs regression tests, safety cases, and cost measurements before it becomes a production dependency.
How to estimate
Start with a request-level cost model, then extend it to monthly usage. Keep model pricing inputs separate from traffic assumptions so the worksheet can be updated without rebuilding the calculation.
1. Estimate the cost of one request
For a model that charges separately for input and output tokens, use:
Request cost = (input tokens ÷ 1,000,000 × input price) + (output tokens ÷ 1,000,000 × output price)
Use the provider's billing unit and pricing terms rather than assuming every service uses the same token interval. If the provider uses a different unit, adapt the denominator accordingly.
2. Estimate monthly model spend
Then calculate:
Monthly model spend = request cost × monthly requests
Do not stop at the average. Calculate at least three scenarios:
- Expected: your best estimate of normal traffic and average token usage.
- High: increased traffic, longer context, more retries, or a seasonal workload.
- Stress: a capacity or abuse scenario that tests rate limits and budget controls.
3. Add non-model costs and operational headroom
Model calls are only one part of the bill. List hosting, databases, vector storage, embedding calls, file storage, queueing, monitoring, network transfer, and any third-party tools. Keep these as separate line items so a change in one service is visible.
Finally, record a contingency allowance for retries, failed requests, evaluation traffic, development environments, and unexpected usage. This is an internal planning assumption, not a guaranteed forecast. The purpose is to prevent an average-case estimate from being treated as a spending limit.
Inputs and assumptions
Create a small versioned worksheet or configuration document with the following fields. Record the date, source, and unit for each input.
Traffic and workload
- Requests per day and per month.
- Peak requests per minute and expected concurrency.
- Percentage of requests that use RAG, tools, file processing, or multi-step workflows.
- Expected retry rate, timeout rate, and fallback rate.
- Estimated cache-hit rate, if responses or retrieval results can be safely cached.
Token and model assumptions
- Average and high-percentile input tokens, including system instructions, conversation history, retrieved text, and tool results.
- Average and maximum output tokens.
- Model selected for each workflow, including fallback models.
- Current input and output pricing, billing units, minimum charges, and any special treatment of cached or batch requests.
- Maximum context size, rate limits, timeout behavior, and availability expectations documented by the provider.
Security and privacy controls
Store provider credentials in a secrets manager or equivalent protected service. Do not place keys in source code, client-side bundles, prompts, tickets, or logs. Give each environment and service only the permissions it needs, rotate credentials, and define a revocation procedure.
Validate input before it reaches the model. Enforce size, type, encoding, and file limits; reject unexpected fields; and scan uploads according to the application's risk profile. Treat retrieved documents, web content, tool output, and user-provided instructions as untrusted data. They may contain text that attempts to override application rules.
Prompt-injection defenses should be layered rather than dependent on a single instruction. Separate instructions from data, constrain tool permissions, validate tool arguments in application code, require confirmation for consequential actions, and keep sensitive operations outside the model's direct control. For RAG workflows, define which sources are allowed and make citations or source identifiers available for review. See these RAG prompt examples and system prompt templates for patterns that can be adapted to your workflow.
Validate outputs as well. Prefer structured responses with a schema, reject malformed or incomplete values, constrain enumerations, and escape generated text before rendering it in a browser or passing it to another system. A model response should be treated as untrusted application data, not executable code.
Reliability and observability
Set explicit timeouts, bounded retries, and backoff. Retrying every failure immediately can increase latency and cost while worsening an upstream incident. Define a fallback for each important workflow: a smaller model, a cached response, a queue for later processing, a deterministic rule, or a clear failure message.
Track request count, success rate, timeout rate, latency by percentile, input and output tokens, estimated cost, retry count, fallback usage, validation failures, and user-visible errors. Use correlation IDs to connect an incoming request with retrieval, tool calls, model calls, and the final response. Avoid logging raw prompts or outputs by default when they may contain personal, confidential, or proprietary information. Redact sensitive fields and define retention periods.
Before launch, establish alerts for budget thresholds, sudden token growth, error-rate changes, latency regression, unusual traffic, repeated validation failures, and unexpected tool activity. Monitoring is useful only when an owner knows what action to take after an alert.
Worked examples
The following examples use illustrative inputs only. They are not current provider prices and should not be used as a purchasing quote.
Example 1: A single model request
Assume a request contains 2,000 input tokens and produces 600 output tokens. For a worksheet, enter the provider's current input price as P-in per million tokens and output price as P-out per million tokens:
Request cost = (2,000 ÷ 1,000,000 × P-in) + (600 ÷ 1,000,000 × P-out)
This makes the cost drivers visible. If conversation history or retrieved documents increase input tokens, the first term rises even when the user's new message is short. If the application permits long answers, the second term becomes more significant.
Example 2: Monthly traffic with retries
Suppose a service expects 30,000 user requests per month. Each request normally produces one model call, but the planning assumption includes a 5% retry rate. Estimated model calls are:
30,000 × 1.05 = 31,500 model calls
If a typical call costs C under the current pricing worksheet, estimated monthly model spend is 31,500 × C. Add separate rows for embedding calls, reranking, tool-related model calls, and evaluation traffic rather than hiding them inside the average.
Example 3: Comparing a prompt optimization
Assume a workflow currently averages 4,000 input tokens and 800 output tokens. A tested change removes repeated instructions and limits retrieved context, reducing the average to 2,500 input tokens while leaving output unchanged. The estimated input-token reduction is:
(4,000 − 2,500) ÷ 4,000 = 37.5%
That percentage describes token reduction, not guaranteed total-cost reduction. Validate whether quality, retrieval coverage, latency, retries, and fallback usage change at the same time. Run the revised prompt against a fixed evaluation set and compare accuracy, safety, consistency, latency, and cost. The LLM prompt testing framework can help organize that comparison.
When to recalculate
Recalculate the production LLM checklist whenever an input to the operating model changes. At minimum, review it after a provider changes pricing, billing units, model availability, context limits, rate limits, or API behavior. Also review it when your application changes its system prompt, conversation memory, retrieval settings, output schema, tool permissions, or fallback logic.
Usage changes are equally important. Recalculate after a launch, marketing event, new customer segment, feature release, or material shift in average prompt and response length. Compare forecast values with measured token usage and latency instead of carrying forward assumptions indefinitely.
Use this practical release sequence:
- Run security tests for authentication, authorization, prompt injection, data leakage, unsafe tool arguments, oversized inputs, and malformed outputs.
- Run prompt and regression tests using representative, adversarial, and previously failed cases.
- Recalculate expected, high, and stress costs using current provider inputs.
- Confirm rate limits, timeouts, retries, fallbacks, queues, and budget alerts in a production-like environment.
- Launch with a controlled exposure, review dashboards and logs, and verify that redaction and retention rules work as intended.
- After the review period, record incidents, cost variance, quality regressions, and changes to the assumptions worksheet.
Keep the checklist, cost model, prompt version, evaluation results, and operational owner together. That record turns production LLM development from a one-time launch exercise into a maintainable engineering process.
