Production LLM Stack: Routing, Evals, Cost, Reliability

A production LLM stack is a control loop around a component that can return a fluent, wrong answer with 200 OK.

The model call is only one box:

client
  → ingress and policy
  → task orchestrator
  → context, retrieval, memory, and tools
  → model gateway and provider
  → output validation
  → product action

every step → trace → evaluation → release decision

The hard part is not connecting another API. It is preserving one product contract while prompts, models, tools, knowledge, traffic, and provider behavior change.

The rest of this guide lays out that operating model. The linked deep dives cover each subsystem without turning this page into a vendor manual.

Start with a task contract

Do not begin with “Which model should we use?” Write down what one user-visible task must do.

For a support-answer task, the contract might be:

task: support.answer
input_schema: support_answer_request.v3
output_schema: support_answer_response.v2
required_capabilities:
  - structured_output
  - tool_calling
allowed_data_regions: [eu]
max_total_latency_ms: 8000
max_provider_attempts: 2
requires_citations: true
requires_human_review_when:
  - refund_amount > 500
  - source_conflict == true

This is not a prompt. It is the application’s promise. It tells the router which models are eligible, the orchestrator when to stop, the validator what to reject, and the evaluation suite what to score.

Keep three kinds of success separate:

A valid JSON object can still be wrong. A good-looking answer can still produce no useful outcome. You need all three layers.

Separate the data plane from the control plane

The data plane handles live requests: load context, call tools and models, validate the result, and return it. Keep this path small and bounded.

The control plane decides what the data plane should run:

• prompt and policy versions;
• eligible models and providers;
• routing and fallback rules;
• evaluation datasets and release thresholds;
• budgets, quotas, and kill switches;
• rollout labels and rollback targets.

Do not let a dashboard edit mutate every production request with no history. Every control-plane change should produce an immutable release identifier. Record that ID on the trace.

One practical release manifest looks like this:

{
  "release": "support-answer-2026-08-21.3",
  "prompt": "support-answer@41",
  "policy": "support-policy@12",
  "router": "support-router@7",
  "retrieval": "help-center-index@2026-08-20",
  "toolset": "support-tools@5",
  "outputSchema": "support_answer_response.v2"
}

That manifest turns “the AI got worse” into a diff you can inspect.

Put policy before routing

Authenticate the caller and classify the request before choosing a provider. The router should receive verified application attributes, not trust fields written inside the user’s prompt.

Useful policy inputs include:

• task and tenant;
• data class and permitted processing regions;
• required capabilities, such as vision or structured output;
• tool permissions;
• latency class and remaining deadline;
• per-user, tenant, and task budget;
• release or experiment assignment.

The result should be an explicit execution envelope:

{
  "task": "support.answer",
  "tenantId": "tenant_internal_42",
  "dataClass": "confidential",
  "allowedProviders": ["provider-eu-a"],
  "allowedTools": ["search_help_center", "read_subscription"],
  "deadlineMs": 8000,
  "release": "support-answer-2026-08-21.3"
}

Do not send raw provider credentials or sensitive classification details to the browser. The server-side execution layer owns both.

Use an internal model contract

Provider SDK types leak quickly: finish reasons, tool-call shapes, usage fields, and error codes end up spread across business logic. Put a narrow adapter between the application and each provider or gateway.

At minimum, normalize:

• request messages and content types;
• structured-output and tool definitions;
• timeout and cancellation;
• response content and tool calls;
• usage and cost inputs;
• provider request ID;
• error class and retryability;
• model and provider actually used.

Do not flatten away capabilities that matter. A “universal” interface with an escape hatch for every provider merely hides lock-in. Keep a common core plus declared capabilities, then reject an incompatible route before the call.

An LLM gateway can centralize this boundary, credentials, quotas, and telemetry. It is not automatically the right first step. Start with one provider behind your own adapter. Add a gateway once more than one service needs the same operational controls. The multi-provider architecture guide covers that layer in detail.

Route tasks, not vague notions of intelligence

The safest router is deterministic first. Filter candidates, then choose among the eligible set:

required modality and tool support
→ data and regional policy
→ output-contract compatibility
→ remaining latency budget
→ quality threshold on this task
→ rate and spend constraints
→ preferred candidate

Cost belongs near the end. A cheap model that fails the task twice is not cheap.

Start with a routing table, not another model call:

An AI router introduces another prompt, latency, cost, and failure mode. Use one only when intent cannot be classified reliably with request metadata or deterministic logic—and evaluate the router separately from the downstream model.

Budget context explicitly

Context is a scarce input, not a bag to fill. Give instructions, current state, retrieval, memory, tool results, and conversation history separate budgets and provenance.

system and policy       fixed, highest priority
task input              required
current structured state exact and scoped
retrieved evidence      ranked, cited, freshness-checked
memory                  filtered by subject and lifecycle
tool results             typed and size-limited
conversation history    summarized only when needed

The context engineering guide covers assembly and priority. The agent memory guide covers durable state. Keep both outside the model adapter so you can test them independently.

For retrieval, log document IDs, index version, filters, scores, and the final chunks sent to the model. If an answer is wrong, you need to distinguish “the source was missing” from “retrieval missed it” and “the model ignored it.”

Treat tools as privileged code paths

A tool call is a proposed action, not authorization. Validate it in application code using the verified principal and current state.

Each tool needs:

• a narrow schema and bounded output;
• server-side authorization;
• a timeout shorter than the task deadline;
• an idempotency key for retried writes;
• an audit event without raw secrets or unnecessary PII;
• an approval boundary for consequential actions;
• a typed error the orchestrator can reason about.

Do not hand the model a catch-all run_sql, http_request, or shell tool in a user- facing workflow. Split capabilities by business action. For remote MCP tools, apply the same controls described in the MCP security guide.

Keep the orchestrator responsible for the loop limit. A model should not be able to spend an unbounded number of tool calls because it keeps “investigating.”

Validate after the model, before the side effect

Structured output moves failure from prose parsing to contract validation, but it does not make the values true.

Use layered validation:

1. Syntactic: parseable JSON or the expected content type.
2. Structural: schema, enum, required fields, length, and cardinality.
3. Referential: IDs exist and belong to the current tenant.
4. Grounding: citations resolve to evidence actually supplied.
5. Policy: the requested action is allowed for this principal and state.
6. Business: totals reconcile, dates are possible, transition is valid.

The validator returns a typed result: accept, repair within the remaining budget, request human review, or fail. Never let free-form model output flow directly into a payment, deletion, permission change, or customer-facing claim.

Design reliability around one deadline

Give the whole task an end-to-end deadline. Every retrieval, tool, model, retry, and validation step spends from it. Independent 30-second timeouts at five layers can turn one request into minutes of waiting.

Classify failures before deciding what to do:

Retries multiply load during an incident. Use a small attempt budget, exponential backoff with jitter, and a circuit breaker to stop sending traffic to a failing dependency.

Cloudflare AI Gateway is one implementation example. Its current request-handling documentation supports per-request timeouts and bounded retry configuration, while the gateway also offers routing, fallback, rate limiting, caching, and budget controls. Those switches still need an application-level deadline and compatibility policy.

Fallback is not “try the next model.” Before a release, prove that each fallback:

• supports the same required tools and output contract;
• may process this data class and region;
• fits the remaining latency and cost budget;
• passes the task dataset at the agreed threshold;
• cannot repeat an already committed side effect.

When no candidate qualifies, degrade honestly: return a partial read-only answer, queue the task, request review, or fail with a stable error.

Trace the user-visible task

Create one root trace for one product task. Put retrieval, tool calls, model generations, parsing, validation, retries, and fallbacks under it.

Use stable operation names:

support.answer
├── load-policy
├── retrieve-help-center
├── generate-answer
├── validate-citations
└── record-outcome

Record release IDs and bounded diagnostic dimensions:

• task, environment, app release, and experiment;
• prompt, policy, router, retrieval, and toolset versions;
• chosen model/provider and fallback reason;
• token usage, estimated cost, and step latency;
• contract result, quality scores, and product outcome;
• internal request IDs needed for correlation and deletion.

Avoid user text, secrets, access tokens, and raw sensitive tool results by default. Sample successful traffic if volume requires it, but retain enough error and rollout coverage to debug regressions.

Langfuse’s current guidance likewise recommends stable trace and observation names, one meaningful root input/output, model and cost metadata on generations, and links between prompt versions and traces. The LLM observability guide turns this into a telemetry contract; the Langfuse setup tutorial covers the implementation.

Build the evaluation loop before model churn

An eval is a release test, not a demo leaderboard. Build it from the tasks your product must handle.

Use several evaluator types:

1. deterministic checks for schemas, citations, permissions, totals, and known labels;
2. reference comparison where a stable expected answer exists;
3. human review for high-impact or ambiguous judgments;
4. a calibrated LLM judge for narrow rubric-based signals at scale;
5. real product outcomes, joined later when they become available.

The LLM-as-judge guide explains where a judge helps and where it lies. The human-in-the-loop guide covers review queues and escalation.

Keep a fixed regression set with IDs, task inputs, policy context, expected invariants, and slice labels. Add adversarial and boundary cases, not only happy paths. Compare the candidate and current release on the same dataset and inspect failures, not just one average score.

Then close the loop:

production failure or reviewed complaint
→ redact and reproduce
→ add or update a dataset case
→ write the cheapest reliable evaluator
→ run current and candidate releases
→ canary
→ monitor the same signal online

Langfuse documents the same offline-to-online cycle in its evaluation concepts: test a change on a fixed dataset, monitor live traces, and feed uncovered edge cases back into the dataset. Tool choice is secondary; the feedback loop is the asset.

Measure cost per successful task

Tokens are an input metric. The useful unit is:

total model + retrieval + tool + evaluation cost
-------------------------------------------------
number of tasks that passed the product success criterion

Segment it by task and release. An inexpensive model with more retries, repairs, and human escalations may raise the real unit cost.

Use four guardrails:

• per-request maximum tokens, tool steps, and total attempts;
• per-user, tenant, task, and environment quotas;
• spend alerts plus a hard-stop or approved degraded route;
• asynchronous processing for work that does not need interactive latency.

Cache only when reuse is semantically safe. Include prompt, model, policy, retrieval version, locale, and relevant user scope in the key. Never share personalized or permission-dependent responses across principals. The LLM cost optimization guide covers prompt caching, semantic caching, batching, and model downsizing with their trade-offs.

Release prompts, models, and retrieval together

Treat these as software releases, even when they live in a dashboard.

A safe path is:

1. create immutable prompt, policy, router, and retrieval versions;
2. run deterministic tests and the fixed evaluation set;
3. compare quality, contract failures, latency, and cost by important slice;
4. canary only traffic eligible for the candidate;
5. watch predefined guardrails and product outcomes;
6. promote a label or roll it back to the previous version;
7. keep the release manifest on every trace.

Langfuse prompt management uses immutable versions and movable labels such as production; its current data-model guide documents label-based promotion and rollback. Cache remote prompt configuration with a known-good local fallback so a control-plane outage does not take down the data plane.

Do not roll out a new prompt, model, retriever, and tool schema simultaneously unless the release must be atomic. Otherwise you lose attribution and make rollback harder.

Add alerts that lead to action

Alert on service and product failure, not every noisy dimension. A small starting set:

• contract failures above the release baseline;
• end-to-end latency or timeout rate over the task SLO;
• fallback and circuit-open rates;
• cost per successful task or spend velocity;
• retrieval-empty and unresolved-citation rates;
• quality score decline on a stable sampled slice;
• human-review backlog age for high-impact tasks.

Every alert needs an owner, a window, a runbook, and a release dimension. The metric alerting guide covers thresholds and burn-rate thinking. Define event names once with the AI event taxonomy guide so dashboards do not become a pile of incompatible counters.

Protect data and credentials by design

Before production traffic, answer:

• Which data classes may reach each provider and region?
• Which fields are removed or tokenized before inference and telemetry?
• How are provider keys stored, scoped, rotated, and audited?
• Can a tenant export or delete its prompts, traces, memory, and cached outputs?
• Which tools can read or mutate which resources?
• What is the retention period for raw inputs, derived scores, and backups?

Use environment variables or a secret manager, not source code or client bundles. Separate staging and production credentials and quotas. OpenAI’s current production guidance also recommends server-side secret storage, separate environments, and explicit spend controls.

Redaction is not a regex you bolt onto logging later. Decide which fields are allowed at the task boundary, carry that classification through the trace, and test that error paths do not log the raw payload.

A rollout sequence that keeps complexity earned

Phase 1: one task, one provider

• typed input and output contracts;
• one server-side provider adapter;
• end-to-end deadline and validation;
• root trace with release identifiers;
• fixed regression dataset;
• explicit spend and tool-step limits.

Phase 2: operational controls

• prompt and policy versioning;
• dashboards and actionable alerts;
• bounded retry and circuit breaker;
• second provider tested as a fallback;
• canary and rollback procedure.

Phase 3: routing and cost

• deterministic task routing;
• per-task quality and cost thresholds;
• safe caching and async/batch paths;
• online sampled evaluation;
• production failures promoted to regression cases.

Phase 4: agents and advanced orchestration

• multiple tools and bounded loops;
• durable memory with lifecycle controls;
• human approval for consequential actions;
• trajectory and tool-behavior evaluation;
• cross-service trace propagation.

Do not start at phase four because an agent framework made the demo easy. Each phase adds states you must test, observe, and recover.

Production readiness checklist

Contract and policy

• Each task has typed inputs, outputs, success criteria, and a total deadline.
• Authentication, data policy, and tool permissions are decided before routing.
• The model cannot authorize its own side effects.

Execution and reliability

• Provider behavior is isolated behind a capability-aware adapter.
• Retries are bounded, idempotent where needed, and share the task deadline.
• Every fallback preserves capabilities, policy, schema, and evaluated quality.
• Partial streams and committed side effects have explicit recovery behavior.

Quality and releases

• Current and candidate releases run on the same representative dataset.
• Contract checks, human review, and calibrated judges cover different failure types.
• Prompts, models, policies, tools, and retrieval versions appear on traces.
• Canary, guardrails, rollback, and a kill switch have been exercised.

Operations and cost

• One root trace represents one user-visible task.
• Dashboards show task success, latency, fallback, and cost per successful task.
• Budgets exist per request and at the correct user, tenant, or task boundary.
• Telemetry failure does not break the product path.

Security and data

• Keys stay server-side and are separated by environment.
• Sensitive inputs and tool results are minimized before inference and telemetry.
• Retention, access, correction, export, and deletion paths are documented and tested.

The stack is ready when a release can fail in a known way, leave enough evidence to explain why, and roll back without guessing. More vendors and more agents do not create that property. Contracts, bounded execution, traces, evals, and disciplined releases do.