# Production LLM Stack: Routing, Evals, Cost, Reliability

> A practical production LLM stack: request contracts, model routing, tools, validation, observability, evals, cost controls, fallbacks, and safe rollouts.
> Author: Roman Belov · Published: 2026-08-21 · Source: https://futurecraft.pro/blog/production-llm-stack/

A production LLM stack is a control loop around a component that can return a fluent,
wrong answer with `200 OK`.

The model call is only one box:

```text
client
  → ingress and policy
  → task orchestrator
  → context, retrieval, memory, and tools
  → model gateway and provider
  → output validation
  → product action

every step → trace → evaluation → release decision
```

The hard part is not connecting another API. It is preserving one product contract
while prompts, models, tools, knowledge, traffic, and provider behavior change.

The rest of this guide lays out that operating model. The linked deep dives cover each
subsystem without turning this page into a vendor manual.

## Start with a task contract

Do not begin with “Which model should we use?” Write down what one user-visible task
must do.

For a support-answer task, the contract might be:

```yaml
task: support.answer
input_schema: support_answer_request.v3
output_schema: support_answer_response.v2
required_capabilities:
  - structured_output
  - tool_calling
allowed_data_regions: [eu]
max_total_latency_ms: 8000
max_provider_attempts: 2
requires_citations: true
requires_human_review_when:
  - refund_amount > 500
  - source_conflict == true
```

This is not a prompt. It is the application's promise. It tells the router which
models are eligible, the orchestrator when to stop, the validator what to reject, and
the evaluation suite what to score.

Keep three kinds of success separate:

| Layer | Question | Example signal |
| --- | --- | --- |
| Contract | Did the output obey the interface? | Schema valid, citation IDs resolve |
| Task quality | Was the result good enough? | Correct label, grounded answer, approved plan |
| Product outcome | Did it help? | Ticket resolved, edit accepted, booking completed |

A valid JSON object can still be wrong. A good-looking answer can still produce no
useful outcome. You need all three layers.

## Separate the data plane from the control plane

The **data plane** handles live requests: load context, call tools and models, validate
the result, and return it. Keep this path small and bounded.

The **control plane** decides what the data plane should run:

- prompt and policy versions;
- eligible models and providers;
- routing and fallback rules;
- evaluation datasets and release thresholds;
- budgets, quotas, and kill switches;
- rollout labels and rollback targets.

Do not let a dashboard edit mutate every production request with no history. Every
control-plane change should produce an immutable release identifier. Record that ID on
the trace.

One practical release manifest looks like this:

```json
{
  "release": "support-answer-2026-08-21.3",
  "prompt": "support-answer@41",
  "policy": "support-policy@12",
  "router": "support-router@7",
  "retrieval": "help-center-index@2026-08-20",
  "toolset": "support-tools@5",
  "outputSchema": "support_answer_response.v2"
}
```

That manifest turns “the AI got worse” into a diff you can inspect.

## Put policy before routing

Authenticate the caller and classify the request before choosing a provider. The
router should receive verified application attributes, not trust fields written inside
the user's prompt.

Useful policy inputs include:

- task and tenant;
- data class and permitted processing regions;
- required capabilities, such as vision or structured output;
- tool permissions;
- latency class and remaining deadline;
- per-user, tenant, and task budget;
- release or experiment assignment.

The result should be an explicit execution envelope:

```json
{
  "task": "support.answer",
  "tenantId": "tenant_internal_42",
  "dataClass": "confidential",
  "allowedProviders": ["provider-eu-a"],
  "allowedTools": ["search_help_center", "read_subscription"],
  "deadlineMs": 8000,
  "release": "support-answer-2026-08-21.3"
}
```

Do not send raw provider credentials or sensitive classification details to the
browser. The server-side execution layer owns both.

## Use an internal model contract

Provider SDK types leak quickly: finish reasons, tool-call shapes, usage fields, and
error codes end up spread across business logic. Put a narrow adapter between the
application and each provider or gateway.

At minimum, normalize:

- request messages and content types;
- structured-output and tool definitions;
- timeout and cancellation;
- response content and tool calls;
- usage and cost inputs;
- provider request ID;
- error class and retryability;
- model and provider actually used.

Do not flatten away capabilities that matter. A “universal” interface with an escape
hatch for every provider merely hides lock-in. Keep a common core plus declared
capabilities, then reject an incompatible route before the call.

An LLM gateway can centralize this boundary, credentials, quotas, and telemetry. It is
not automatically the right first step. Start with one provider behind your own
adapter. Add a gateway once more than one service needs the same operational controls.
The [multi-provider architecture guide](/blog/multi-provider-llm-architecture/) covers
that layer in detail.

## Route tasks, not vague notions of intelligence

The safest router is deterministic first. Filter candidates, then choose among the
eligible set:

```text
required modality and tool support
→ data and regional policy
→ output-contract compatibility
→ remaining latency budget
→ quality threshold on this task
→ rate and spend constraints
→ preferred candidate
```

Cost belongs near the end. A cheap model that fails the task twice is not cheap.

Start with a routing table, not another model call:

| Task class | Required behavior | Default | Allowed fallback |
| --- | --- | --- | --- |
| Intent classification | Fixed enum, low latency | Small validated model | Rules or same-class model |
| Grounded support answer | Citations and retrieval | Model proven on support set | Equivalent grounded model |
| Document extraction | Strict schema | Structured-output model | Async review queue |
| High-impact action | Tool call plus approval | Capable model | No silent fallback |

An AI router introduces another prompt, latency, cost, and failure mode. Use one only
when intent cannot be classified reliably with request metadata or deterministic
logic—and evaluate the router separately from the downstream model.

## Budget context explicitly

Context is a scarce input, not a bag to fill. Give instructions, current state,
retrieval, memory, tool results, and conversation history separate budgets and
provenance.

```text
system and policy       fixed, highest priority
task input              required
current structured state exact and scoped
retrieved evidence      ranked, cited, freshness-checked
memory                  filtered by subject and lifecycle
tool results             typed and size-limited
conversation history    summarized only when needed
```

The [context engineering guide](/blog/context-engineering-guide/) covers assembly and
priority. The [agent memory guide](/blog/ai-agent-memory/) covers durable state. Keep
both outside the model adapter so you can test them independently.

For retrieval, log document IDs, index version, filters, scores, and the final chunks
sent to the model. If an answer is wrong, you need to distinguish “the source was
missing” from “retrieval missed it” and “the model ignored it.”

## Treat tools as privileged code paths

A tool call is a proposed action, not authorization. Validate it in application code
using the verified principal and current state.

Each tool needs:

- a narrow schema and bounded output;
- server-side authorization;
- a timeout shorter than the task deadline;
- an idempotency key for retried writes;
- an audit event without raw secrets or unnecessary PII;
- an approval boundary for consequential actions;
- a typed error the orchestrator can reason about.

Do not hand the model a catch-all `run_sql`, `http_request`, or shell tool in a user-
facing workflow. Split capabilities by business action. For remote MCP tools, apply
the same controls described in the [MCP security guide](/blog/mcp-security-guide/).

Keep the orchestrator responsible for the loop limit. A model should not be able to
spend an unbounded number of tool calls because it keeps “investigating.”

## Validate after the model, before the side effect

Structured output moves failure from prose parsing to contract validation, but it does
not make the values true.

Use layered validation:

1. **Syntactic:** parseable JSON or the expected content type.
2. **Structural:** schema, enum, required fields, length, and cardinality.
3. **Referential:** IDs exist and belong to the current tenant.
4. **Grounding:** citations resolve to evidence actually supplied.
5. **Policy:** the requested action is allowed for this principal and state.
6. **Business:** totals reconcile, dates are possible, transition is valid.

The validator returns a typed result: accept, repair within the remaining budget,
request human review, or fail. Never let free-form model output flow directly into a
payment, deletion, permission change, or customer-facing claim.

## Design reliability around one deadline

Give the whole task an end-to-end deadline. Every retrieval, tool, model, retry, and
validation step spends from it. Independent 30-second timeouts at five layers can turn
one request into minutes of waiting.

Classify failures before deciding what to do:

| Failure | Retry same route? | Fallback? | Typical action |
| --- | --- | --- | --- |
| Timeout before output | Once, if budget remains | Yes, compatible route | Backoff with jitter |
| Provider 429/5xx | Bounded | Yes | Respect retry hints, open circuit on sustained failure |
| Invalid request/schema | No | Usually no | Fix caller or contract |
| Authentication/permission | No | No | Stop and alert |
| Output contract failure | Maybe one repair | Only evaluated equivalent | Validate again |
| Partial stream | No transparent replay | Explicit restart only | Tell client and avoid duplicate side effects |

Retries multiply load during an incident. Use a small attempt budget, exponential
backoff with jitter, and a [circuit breaker](/blog/circuit-breaker-deno-edge-functions/)
to stop sending traffic to a failing dependency.

Cloudflare AI Gateway is one implementation example. Its current
[request-handling documentation](https://developers.cloudflare.com/ai-gateway/configuration/request-handling/)
supports per-request timeouts and bounded retry configuration, while the gateway also
offers routing, fallback, rate limiting, caching, and budget controls. Those switches
still need an application-level deadline and compatibility policy.

Fallback is not “try the next model.” Before a release, prove that each fallback:

- supports the same required tools and output contract;
- may process this data class and region;
- fits the remaining latency and cost budget;
- passes the task dataset at the agreed threshold;
- cannot repeat an already committed side effect.

When no candidate qualifies, degrade honestly: return a partial read-only answer,
queue the task, request review, or fail with a stable error.

## Trace the user-visible task

Create one root trace for one product task. Put retrieval, tool calls, model
generations, parsing, validation, retries, and fallbacks under it.

Use stable operation names:

```text
support.answer
├── load-policy
├── retrieve-help-center
├── generate-answer
├── validate-citations
└── record-outcome
```

Record release IDs and bounded diagnostic dimensions:

- task, environment, app release, and experiment;
- prompt, policy, router, retrieval, and toolset versions;
- chosen model/provider and fallback reason;
- token usage, estimated cost, and step latency;
- contract result, quality scores, and product outcome;
- internal request IDs needed for correlation and deletion.

Avoid user text, secrets, access tokens, and raw sensitive tool results by default.
Sample successful traffic if volume requires it, but retain enough error and rollout
coverage to debug regressions.

Langfuse's current guidance likewise recommends stable trace and observation names,
one meaningful root input/output, model and cost metadata on generations, and links
between prompt versions and traces. The
[LLM observability guide](/blog/llm-observability-langfuse/) turns this into a telemetry
contract; the [Langfuse setup tutorial](/blog/langfuse-step-by-step/) covers the
implementation.

## Build the evaluation loop before model churn

An eval is a release test, not a demo leaderboard. Build it from the tasks your product
must handle.

Use several evaluator types:

1. deterministic checks for schemas, citations, permissions, totals, and known labels;
2. reference comparison where a stable expected answer exists;
3. human review for high-impact or ambiguous judgments;
4. a calibrated LLM judge for narrow rubric-based signals at scale;
5. real product outcomes, joined later when they become available.

The [LLM-as-judge guide](/blog/llm-as-judge-automated-quality-gate/) explains where a
judge helps and where it lies. The [human-in-the-loop guide](/blog/human-in-the-loop/)
covers review queues and escalation.

Keep a fixed regression set with IDs, task inputs, policy context, expected invariants,
and slice labels. Add adversarial and boundary cases, not only happy paths. Compare the
candidate and current release on the same dataset and inspect failures, not just one
average score.

Then close the loop:

```text
production failure or reviewed complaint
→ redact and reproduce
→ add or update a dataset case
→ write the cheapest reliable evaluator
→ run current and candidate releases
→ canary
→ monitor the same signal online
```

Langfuse documents the same offline-to-online cycle in its
[evaluation concepts](https://langfuse.com/docs/evaluation/core-concepts): test a
change on a fixed dataset, monitor live traces, and feed uncovered edge cases back into
the dataset. Tool choice is secondary; the feedback loop is the asset.

## Measure cost per successful task

Tokens are an input metric. The useful unit is:

```text
total model + retrieval + tool + evaluation cost
-------------------------------------------------
number of tasks that passed the product success criterion
```

Segment it by task and release. An inexpensive model with more retries, repairs, and
human escalations may raise the real unit cost.

Use four guardrails:

- per-request maximum tokens, tool steps, and total attempts;
- per-user, tenant, task, and environment quotas;
- spend alerts plus a hard-stop or approved degraded route;
- asynchronous processing for work that does not need interactive latency.

Cache only when reuse is semantically safe. Include prompt, model, policy, retrieval
version, locale, and relevant user scope in the key. Never share personalized or
permission-dependent responses across principals. The
[LLM cost optimization guide](/blog/ai-cost-optimization/) covers prompt caching,
semantic caching, batching, and model downsizing with their trade-offs.

## Release prompts, models, and retrieval together

Treat these as software releases, even when they live in a dashboard.

A safe path is:

1. create immutable prompt, policy, router, and retrieval versions;
2. run deterministic tests and the fixed evaluation set;
3. compare quality, contract failures, latency, and cost by important slice;
4. canary only traffic eligible for the candidate;
5. watch predefined guardrails and product outcomes;
6. promote a label or roll it back to the previous version;
7. keep the release manifest on every trace.

Langfuse prompt management uses immutable versions and movable labels such as
`production`; its current
[data-model guide](https://langfuse.com/docs/prompt-management/data-model) documents
label-based promotion and rollback. Cache remote prompt configuration with a known-good
local fallback so a control-plane outage does not take down the data plane.

Do not roll out a new prompt, model, retriever, and tool schema simultaneously unless
the release must be atomic. Otherwise you lose attribution and make rollback harder.

## Add alerts that lead to action

Alert on service and product failure, not every noisy dimension. A small starting set:

- contract failures above the release baseline;
- end-to-end latency or timeout rate over the task SLO;
- fallback and circuit-open rates;
- cost per successful task or spend velocity;
- retrieval-empty and unresolved-citation rates;
- quality score decline on a stable sampled slice;
- human-review backlog age for high-impact tasks.

Every alert needs an owner, a window, a runbook, and a release dimension. The
[metric alerting guide](/blog/automated-metric-alerts/) covers thresholds and burn-rate
thinking. Define event names once with the
[AI event taxonomy guide](/blog/event-taxonomy-ai/) so dashboards do not become a pile
of incompatible counters.

## Protect data and credentials by design

Before production traffic, answer:

- Which data classes may reach each provider and region?
- Which fields are removed or tokenized before inference and telemetry?
- How are provider keys stored, scoped, rotated, and audited?
- Can a tenant export or delete its prompts, traces, memory, and cached outputs?
- Which tools can read or mutate which resources?
- What is the retention period for raw inputs, derived scores, and backups?

Use environment variables or a secret manager, not source code or client bundles.
Separate staging and production credentials and quotas. OpenAI's current
[production guidance](https://developers.openai.com/api/docs/guides/production-best-practices)
also recommends server-side secret storage, separate environments, and explicit spend
controls.

Redaction is not a regex you bolt onto logging later. Decide which fields are allowed
at the task boundary, carry that classification through the trace, and test that error
paths do not log the raw payload.

## A rollout sequence that keeps complexity earned

### Phase 1: one task, one provider

- typed input and output contracts;
- one server-side provider adapter;
- end-to-end deadline and validation;
- root trace with release identifiers;
- fixed regression dataset;
- explicit spend and tool-step limits.

### Phase 2: operational controls

- prompt and policy versioning;
- dashboards and actionable alerts;
- bounded retry and circuit breaker;
- second provider tested as a fallback;
- canary and rollback procedure.

### Phase 3: routing and cost

- deterministic task routing;
- per-task quality and cost thresholds;
- safe caching and async/batch paths;
- online sampled evaluation;
- production failures promoted to regression cases.

### Phase 4: agents and advanced orchestration

- multiple tools and bounded loops;
- durable memory with lifecycle controls;
- human approval for consequential actions;
- trajectory and tool-behavior evaluation;
- cross-service trace propagation.

Do not start at phase four because an agent framework made the demo easy. Each phase
adds states you must test, observe, and recover.

## Production readiness checklist

**Contract and policy**

- [ ] Each task has typed inputs, outputs, success criteria, and a total deadline.
- [ ] Authentication, data policy, and tool permissions are decided before routing.
- [ ] The model cannot authorize its own side effects.

**Execution and reliability**

- [ ] Provider behavior is isolated behind a capability-aware adapter.
- [ ] Retries are bounded, idempotent where needed, and share the task deadline.
- [ ] Every fallback preserves capabilities, policy, schema, and evaluated quality.
- [ ] Partial streams and committed side effects have explicit recovery behavior.

**Quality and releases**

- [ ] Current and candidate releases run on the same representative dataset.
- [ ] Contract checks, human review, and calibrated judges cover different failure types.
- [ ] Prompts, models, policies, tools, and retrieval versions appear on traces.
- [ ] Canary, guardrails, rollback, and a kill switch have been exercised.

**Operations and cost**

- [ ] One root trace represents one user-visible task.
- [ ] Dashboards show task success, latency, fallback, and cost per successful task.
- [ ] Budgets exist per request and at the correct user, tenant, or task boundary.
- [ ] Telemetry failure does not break the product path.

**Security and data**

- [ ] Keys stay server-side and are separated by environment.
- [ ] Sensitive inputs and tool results are minimized before inference and telemetry.
- [ ] Retention, access, correction, export, and deletion paths are documented and tested.

The stack is ready when a release can fail in a known way, leave enough evidence to
explain why, and roll back without guessing. More vendors and more agents do not create
that property. Contracts, bounded execution, traces, evals, and disciplined releases
do.