Model fallback

Model fallback is the practice of automatically rerouting a request to a designated alternative model when the primary model refuses, errors, or is unavailable, a per-request runtime decision executed by gateway or vendor machinery, as distinct from the planned, workload-level version change of a model migration.

How it works

Two trigger families share one pattern. Availability-triggered fallback is the established gateway practice: when the primary model rate-limits, errors, or times out, the request retries down a configured chain of alternatives, and gateway tooling has long shipped this with distinct fallback types for general errors, content-policy declines, and context-window overflows. Refusal-triggered fallback is the newer, first-party variant: when a safety classifier declines a request, the vendor's own API can rerun it on a permitted alternative model, declared through a request parameter or wrapped in SDK middleware, while consumer surfaces switch automatically with a visible notice. In both families a well-built route records which model answered, often as a marker the response itself carries, and follow-up turns can pin to the model that accepted so a conversation does not thrash between models. The reroute is scoped to the request; the workload's primary model assignment does not change. Which alternative models are permitted can be constrained by the vendor rather than chosen freely.

Why it matters

Fallback turns the availability of an answer from a property of one model into a property of the route, which is what production wants on routes whose contract allows an alternative answer, since outages are inevitable and refusals arrive success-shaped. The price is a second model in the dependency graph: its contract, behavior, and pricing apply on precisely the requests where the primary declined, which are rarely the typical ones, and a team migrating away from a model can find it still answering their hardest traffic as the designated understudy. Unobserved fallback also eats signal: a climbing fallback rate, read with its trigger mix, is migration-grade evidence about whether the primary still fits the workload, and a fleet that quietly answers from the fallback invalidates evals and cost baselines pinned to the primary. The distinction from migration is the durable part: fallback absorbs individual requests, migration moves the workload, and the measured fallback rate is often exactly the data that justifies the migration decision. A fallback policy is therefore as much an observability commitment as a resilience feature.

In practice

Mid-pipeline, a request trips the primary model's safety classifier. Without a fallback policy the pipeline holds a declined success response it never planned for; with one, and where policy permits an alternative answer, the request reruns on the designated alternative, the response is labeled with the model that answered, and the conversation stays pinned there for continuity. The dashboard counts the reroute under refusal-triggered fallback, separate from the availability counter. When one route's refusal-fallback count climbs week over week, the team re-evaluates that route's model assignment with data instead of discovering the drift in a quarterly bill.

Practical considerations

Coverage is uneven across surfaces: a first-party fallback parameter may exist on the vendor's own API but not on batch endpoints or every cloud platform, which fall back to SDK middleware in supported languages or hand-rolled retries everywhere else. Log which model answered every request; without that, the fleet can quietly live on the understudy while evals and baselines keep describing the primary. Count refusal-triggered and availability-triggered fires separately, because one is a content decision with a policy fix and the other is an operations event with an infrastructure fix. A fallback adds a retry's latency and a second model's cost on exactly the flagged requests, so latency-sensitive paths should decide in advance whether to fall back or fail fast, and every chain needs an exhaustion rule for the day the final permitted target declines too. Billing for a declined attempt versus its rerun differs by vendor and by whether output had begun streaming, a contract detail to confirm rather than assume. Where the vendor constrains permitted fallback targets, treat the constraint as part of the dependency decision, since it can keep a model you planned to retire inside your graph.

Defined by Ready Solutions AI