Multi-Provider LLM Routing: A Production Architecture

Multi-provider LLM routing is not a list that says “try model B if model A fails.” It is a compatibility and policy layer between an application task and several changing APIs.

The useful question is not “which provider is best?” It is:

Which providers are eligible for this task, and what must remain true if the route changes?

This guide covers that provider and gateway layer. The production LLM stack article connects it to prompt management, evaluation, observability, and release control.

Do You Need a Second Provider?

Do not begin with multi-provider infrastructure by default. It creates a real maintenance surface:

• different request and streaming semantics;
• different tool-calling and structured-output behavior;
• another set of credentials and data-processing terms;
• a larger evaluation matrix;
• more ambiguous incidents.

Start with one provider behind an internal adapter. Add a second provider when one of these risks is both material and measured:

• the task cannot tolerate the observed outage window;
• capacity or rate limits block expected traffic;
• a region or data-handling requirement excludes the primary route;
• one task has a proven quality-to-cost advantage elsewhere;
• a provider or model retirement would take too long to absorb.

If a queued job can wait ten minutes, delayed retry may be safer than switching models. If a response authorizes a payment or changes customer data, returning no answer may be safer than accepting a semantically different fallback.

Begin With a Task Contract

Applications should ask for a task, not a vendor model. A task registry can hold the requirements that routing is allowed to use:

interface TaskPolicy {
  task: 'support_reply' | 'invoice_extract' | 'trip_plan';
  requiredCapabilities: Array<'tools' | 'json_schema' | 'vision'>;
  allowedRegions: string[];
  maxInputTokens: number;
  deadlineMs: number;
  maxAttempts: number;
  outputSchema: string;
  evaluationSuite: string;
}

Keep commercial model names out of product code. A deployment registry maps stable internal names to provider deployments:

“Cheapest available model” is not an eligibility rule. Neither is “fastest this minute.” A route must first satisfy capability, security, context, and quality constraints. Cost and latency can choose among the remaining candidates.

Normalize the Smallest Useful Contract

A gateway should hide provider transport details, not pretend that every model has identical behavior. Normalize only what the application can rely on:

interface LlmRequest {
  requestId: string;
  task: TaskPolicy['task'];
  messages: Array<{ role: 'system' | 'user' | 'assistant'; content: string }>;
  responseFormat: { type: 'text' } | { type: 'json'; schema: string };
  stream: boolean;
}

interface LlmResult {
  text: string;
  provider: string;
  model: string;
  routeReason: string;
  attempt: number;
  inputTokens?: number;
  outputTokens?: number;
}

Provider-specific features should be explicit capabilities. Silently dropping a reasoning, caching, or tool parameter during fallback is a correctness bug. If a task requires that feature, a route without it is ineligible.

Use a stable internal error taxonomy too:

• invalid_request — the application sent a bad request;
• authentication — credentials or permissions are wrong;
• policy_rejection — the provider rejected the content;
• rate_limited — capacity may recover later;
• provider_unavailable — connection or server failure;
• deadline_exceeded — the task budget is spent;
• output_invalid — the response failed the application contract.

This classification matters more than the original HTTP status. Providers do not use identical error bodies, and not every 4xx or 5xx deserves the same action.

Make Routing Deterministic

A production route should be explainable after the fact. One practical order is:

1. Load the task policy and the current route version.
2. Remove deployments that lack a required capability.
3. Apply region, data-retention, and tenant restrictions.
4. Remove deployments that cannot fit the input or remaining deadline.
5. Remove models that have not passed the task’s evaluation gate.
6. Apply rollout percentage and circuit-breaker state.
7. Choose by the declared policy: priority, measured latency, or cost ceiling.

Record the route version and reason on every call. “The router picked it” is not enough during an incident.

Managed gateways can implement part of this policy. For example, Cloudflare AI Gateway documents versioned dynamic routes with conditional, percentage, model, rate, and budget nodes. Treat such a route as production configuration: review changes, promote versions deliberately, and keep a rollback target.

Retry and Fallback Are Different Decisions

A retry repeats a request against the same deployment. A fallback changes the deployment and may change model behavior. Keep the decisions separate.

All attempts share one end-to-end deadline. Three providers with 30-second timeouts do not create a 30-second service; they create a 90-second worst case.

async function runTask(
  request: LlmRequest,
  policy: TaskPolicy,
  candidates: ProviderAdapter[],
): Promise<LlmResult> {
  const startedAt = Date.now();
  const attempts = candidates.slice(0, policy.maxAttempts);

  for (const [index, adapter] of attempts.entries()) {
    const remainingMs = policy.deadlineMs - (Date.now() - startedAt);
    if (remainingMs <= 0) throw new Error('deadline_exceeded');

    try {
      const result = await adapter.complete(request, remainingMs);
      await validateOutput(result.text, policy.outputSchema);
      return { ...result, attempt: index + 1 };
    } catch (error: unknown) {
      const kind = classifyProviderError(error);
      if (!isFallbackEligible(kind)) throw error;
    }
  }

  throw new Error('provider_unavailable');
}

The sketch omits logging and abort signals for clarity. In real code, cancel an expired fetch with AbortSignal, carry a request ID through every attempt, and never log raw prompts containing personal data.

Circuit Breakers Prevent Retry Storms

If a provider is failing, sending every new request through the same timeout only makes recovery harder. A circuit breaker removes a deployment from routing after a defined failure window. After a cool-down, limited probe traffic tests it again.

Scope the breaker narrowly. A failure in one model, region, or credential pool should not disable every call to that provider. It should also distinguish provider failures from bugs in your own request construction.

Use thresholds derived from the service-level objective and traffic volume, not copied percentages. At low traffic, five failures may be more informative than a one-minute error rate. At high traffic, a rolling rate with a minimum sample size is usually steadier.

Streaming Changes the Failure Contract

Before the first response byte, a gateway can discard a failed attempt and try another eligible deployment. After bytes reach the client, a transparent switch is no longer safe: the second model does not know the exact hidden state of the first generation.

Choose an application behavior in advance:

• mark the answer as interrupted and offer “try again”;
• restart the entire response with a new generation ID;
• buffer the complete model response before sending it for critical tasks;
• checkpoint an application workflow between non-streaming steps.

Do not concatenate a second provider’s output onto a broken stream and call it recovery. Tool calls are especially risky: the client may already have executed an action before the connection failed.

Protect Side Effects With Idempotency

An LLM request can finish at the provider even when your gateway times out. A blind retry may therefore generate twice, charge twice, or execute a tool twice.

Give each logical operation an idempotency key in your application. Store tool execution state separately from model generation. Before executing a proposed action, check whether that operation has already completed. The model is allowed to repeat text; your payment, email, or database mutation is not.

For agent workflows, make the boundary explicit:

1. model proposes a typed action;
2. application validates policy and arguments;
3. idempotency store reserves the operation;
4. application executes the tool;
5. result is recorded before the next model step.

Validate Every Fallback Output

Transport compatibility is not semantic compatibility. Two models may accept the same JSON schema and still differ on omitted fields, tool selection, citations, or refusal behavior.

Run the same postconditions regardless of route:

• parse and validate structured output;
• enforce domain constraints beyond JSON shape;
• reject unknown tool names and arguments;
• verify required citations or source identifiers;
• apply deterministic safety and permission checks;
• score sampled outputs against the task evaluation set.

A model becomes eligible only after it passes the task’s offline evaluation and canary criteria. The AI agent testing guide covers the evaluation layer in more detail.

Keep Data Policy at the Gateway

Central routing also centralizes risk. The gateway sees prompts, attachments, tenant identifiers, and provider credentials.

At minimum:

• store provider keys in a secret manager, never in client code or route files;
• allow providers per tenant and data class;
• redact or tokenize sensitive fields before the provider call when possible;
• define log sampling and retention explicitly;
• separate customer-visible request IDs from internal trace data;
• prevent a fallback from crossing a prohibited region or retention policy;
• audit route and credential changes.

Provider fallback must not become compliance fallback.

Observe the Route Decision and the Call

Latency and error rate are necessary but insufficient. Record:

• task and task-policy version;
• route version and route reason;
• attempted and selected deployment;
• actual provider and model returned by the adapter;
• time spent per attempt and total deadline consumed;
• fallback step and normalized error kind;
• input and output tokens when available;
• estimated cost using a versioned price table;
• schema and evaluation outcome;
• prompt or workflow version, without raw personal data.

Alert on sustained changes relative to a baseline: exhausted deadlines, open circuits, fallback share, invalid outputs, and cost per successful task. A low provider error rate is not reassuring if fallback outputs fail the contract.

For trace design, see the LLM observability guide.

Choose the Smallest Control Layer That Works

There are three common implementation options.

An in-app adapter is often enough for the first two providers. A managed gateway fits teams that want centralized policy without operating proxy infrastructure. A self-hosted proxy such as LiteLLM can make sense when several services need one compatibility layer, but the proxy itself becomes production infrastructure. Its failure domain, upgrades, and authentication require the same care as any other gateway.

Avoid relying on a deprecated integration path. Cloudflare, for example, now directs new routing integrations toward Dynamic Routing rather than its older Universal Endpoint fallback format. Check current product documentation before copying gateway configuration from an older tutorial.

Roll Out a Second Provider Without Surprises

Use a staged migration:

1. Extract an adapter. Keep behavior unchanged behind a stable internal API.
2. Create a task evaluation set. Include normal, boundary, refusal, and tool cases from real failure modes.
3. Integrate the candidate with no live traffic. Normalize errors and output.
4. Shadow eligible requests. Do not expose or execute shadow results.
5. Compare contract pass rate, latency, and cost per successful task. Average token price alone is not a decision metric.
6. Canary a small, reversible cohort. Keep the primary route available.
7. Enable bounded fallback for one task. Watch total deadline and invalid output rate.
8. Promote or roll back a versioned route. Record why.

Shadow traffic can duplicate sensitive data and provider cost. Apply the same consent, retention, and regional rules as production traffic.

Production Checklist

• Product code calls stable task names, not provider model names.
• Every task has capabilities, region, deadline, attempt, and output rules.
• Provider-specific features are declared, not silently discarded.
• Retries and fallbacks use a normalized error taxonomy.
• All attempts share one end-to-end deadline and support cancellation.
• Circuit breakers are scoped by deployment and ignore caller errors.
• Streaming interruption has an explicit client experience.
• Tool side effects use application-level idempotency.
• Every eligible model passed the same task evaluation suite.
• Fallback cannot bypass data, safety, or tenant policy.
• Route versions, reasons, attempts, and output validity are observable.
• A tested rollback route exists.

Primary References

• Cloudflare AI Gateway: Dynamic Routing
• Cloudflare AI Gateway: request retries and timeouts
• LiteLLM routing documentation
• AWS Prescriptive Guidance: circuit breaker pattern

The architecture is working when changing a provider does not change the task’s contract, violate its policy, or turn one transient error into a chain of unbounded attempts. Everything else is implementation detail.