LLM API Migration Checklist: How to Switch Models or Providers Without Breaking Your App

Overview

An LLM migration can affect every layer of an application: authentication, request construction, prompt behavior, token accounting, tool execution, streaming, error handling, monitoring, and cost controls. A model that appears compatible during a basic chat test may still fail when it receives long context, produces structured output, invokes a tool, or encounters a rate limit.

Start by defining the migration boundary. Are you replacing one model with another through the same provider, moving between providers, changing an SDK, or redesigning the application around a new capability? The larger the boundary, the more important it is to separate transport compatibility from behavioral compatibility.

Use a provider-neutral adapter wherever practical. Your application should call an internal interface such as generate(), stream(), or embed() rather than spreading provider-specific request formats throughout business logic. Keep provider configuration, authentication, retry behavior, and response normalization behind that boundary.

Before migration, record a baseline using representative inputs. Capture latency, failure rates, output validity, token usage, tool-call success, retrieval quality where relevant, and human-rated answer quality. The baseline gives the team something to compare against instead of relying on a few anecdotal prompts.

Checklist by scenario

Scenario 1: Replacing a model within the same provider

• Confirm the replacement model's identifier, lifecycle status, supported modalities, context limits, and output constraints from the provider's current documentation.
• Check whether the model accepts the same message roles, parameter names, response formats, and tool definitions.
• Re-run prompts that depend on exact formatting, classification labels, refusal behavior, or deterministic output.
• Compare latency, output length, token consumption, and failure modes under realistic concurrency.
• Use a feature flag so a small, observable portion of traffic can use the replacement before a full cutover.

Scenario 2: Moving to a different API provider

• Create credentials with the minimum permissions required, store them in a secret manager, and keep old and new credentials separately rotatable.
• Map the request format explicitly. Do not assume that equivalent concepts use equivalent field names or nesting.
• Normalize provider-specific responses into your application schema, including text, finish reasons, usage, citations, tool calls, and errors.
• Document differences in rate limits, retry headers, timeout behavior, regional routing, data handling options, and service-level controls as applicable to your environment.
• Run contract, load, security, and behavioral tests before sending production traffic.

Scenario 3: Migrating a structured-output or tool-using workflow

• Test whether the target supports the structured-output mechanism your application relies on, rather than treating ordinary text that looks like JSON as equivalent.
• Validate schemas, required fields, enum values, additional properties, and recovery behavior for malformed output.
• Compare tool-call names, argument serialization, parallel-call behavior, refusal handling, and the model's tendency to call tools unnecessarily.
• Keep tool execution behind application-side authorization and validation. The model should propose an action; your application should decide whether it is allowed.
• Test interrupted streams and duplicate events so a partial response cannot trigger an unsafe or repeated side effect.

Capability comparison table

What to double-check

Prompts and message precedence: A prompt can produce different results even when the visible text is unchanged if roles, defaults, safety settings, or instruction precedence change. Version system prompts, few-shot examples, retrieval templates, and tool descriptions. Re-test workflows that depend on exact labels or formatting. For background, see the structured output prompting guide and few-shot classification guide.

Context construction: Migration bugs often appear before generation begins. Check how conversation history, retrieved documents, metadata, and tool results are counted and truncated. A different tokenizer or content representation can change which information reaches the model. For retrieval workflows, compare citation presence, source selection, and answers with missing or conflicting context. The RAG prompt examples provide useful patterns for this review.

Parsing and side effects: Treat model output as untrusted input. Validate JSON, enforce length and type constraints, escape rendered text, and require application-level authorization for external actions. Tool calls should be idempotent where possible, with an operation identifier that prevents accidental repetition.

Resilience: Test timeouts, rate limits, overloaded responses, malformed payloads, empty completions, cancelled streams, and provider outages. A retry is appropriate only when repeating the request is safe. For non-idempotent tools, separate generation from execution and persist execution state.

Observability: Record the provider, model version or identifier, request class, latency, status, token usage when available, schema-validation result, and retry count. Avoid logging secrets and unnecessary sensitive user content. Compare aggregate behavior as well as individual traces. The LLM prompt testing framework can help organize evaluation cases.

Compatibility test template

Test case:
Input class:
Expected behavior:
Required output schema:
Provider/model under test:
Context size:
Tool calls expected: yes/no
Streaming expected: yes/no
Baseline result:
Migration result:
Pass criteria:
Observed differences:
Severity and owner:

Build the test set from production-shaped examples, edge cases, adversarial inputs, long-context requests, empty or missing fields, and previously failed conversations. Compare both pass/fail outcomes and meaningful differences in content, cost, latency, and safety.

Common mistakes

• Changing the provider and prompt at the same time: Keep variables isolated where possible. Otherwise, it becomes difficult to identify why behavior changed.
• Testing only a happy-path chat: Include streaming, tool use, structured output, long context, retries, and malformed responses.
• Assuming model names imply equivalent behavior: Capability labels do not guarantee identical instruction following, formatting, or latency.
• Hard-coding provider responses: Provider-specific fields should be translated at the adapter boundary, not required throughout the application.
• Ignoring cost after launch: Measure actual input and output patterns, including retries, long conversations, and retrieval payloads. See the token cost optimization guide.
• Removing the old integration too early: Keep a tested rollback path until the new integration has passed its observation period and operational review.
• Relying on subjective review alone: Combine automated checks, representative evaluations, trace inspection, and human review for high-impact workflows.

Rollback checklist

• Keep the previous model or provider configuration available and tested.
• Make routing reversible through configuration or a feature flag.
• Preserve compatible prompt and schema versions for the fallback path.
• Define rollback triggers, such as elevated error rate, invalid output, unacceptable latency, or quality regression.
• Confirm that old credentials, quotas, dashboards, and alert routes still work.
• Record the migration state so in-flight jobs are not processed twice.

When to revisit

Revisit this checklist before every provider switch, model replacement, SDK upgrade, scheduled deprecation, major prompt change, or compatibility regression review. It is also worth reviewing before seasonal planning cycles when traffic, data volume, or reliability requirements may change.

After launch, schedule a post-migration review rather than treating cutover as the finish line. Compare the baseline with production observations and answer:

• Did task quality, output validity, latency, or failure behavior change?
• Did input construction or context truncation change?
• Are retries, timeouts, and rate limits behaving as expected?
• Did actual usage and cost assumptions hold?
• Are dashboards, alerts, audit records, and privacy controls complete?
• Can the team roll back without a code deployment?

Close the review by updating the adapter contract, compatibility tests, runbook, model inventory, and ownership information. A migration is complete when the new integration is not only working, but also measurable, reversible, and maintainable. For the broader production checklist, see Deploying an LLM App to Production.