# LLM Cost Optimization: Measure Cost per Successful Task

> Reduce LLM spend without hiding quality loss: attribute costs by task, remove waste, set output budgets, test smaller models, use caching safely, and control retries.
> Author: Roman Belov · Published: 2026-06-09 · Source: https://futurecraft.pro/blog/ai-cost-optimization/

“Cut LLM cost by 60%” is not an engineering target unless the baseline, traffic,
and quality bar are stated. A cheaper request that fails twice, reaches a human,
or damages conversion can cost more than the original.

Use this metric instead:

```text
cost per successful task =
  (model + embeddings + reranking + retries + cache + review + infrastructure)
  / tasks that pass the product outcome and quality contract
```

The [production LLM stack](/blog/production-llm-stack/) guide defines that task
contract. This article focuses on the cost controls inside it.

## Build a Cost Ledger First

Provider invoices aggregate spend. They rarely explain which product decision
created it. Record one ledger row per logical task:

```typescript
interface LlmCostEvent {
  taskId: string;
  taskType: string;
  tenantId: string;
  promptVersion: string;
  routeVersion: string;
  provider: string;
  model: string;
  attempt: number;
  inputTokens?: number;
  cachedInputTokens?: number;
  outputTokens?: number;
  estimatedCost: number;
  latencyMs: number;
  outcome: 'passed' | 'failed' | 'escalated' | 'abandoned';
}
```

Keep price tables versioned by provider, model, region, service tier, and date.
Reconcile estimates with the invoice. Do not log raw prompts or personal data just
to understand cost.

Break dashboards down by task and outcome:

- total and marginal spend;
- cost per passed task;
- tokens per section: policy, retrieval, history, tools, output;
- attempts and fallback share;
- cache read/write tokens and hit rate;
- quality pass, escalation, and abandonment rates;
- P50/P95 latency;
- spend by tenant with budget alerts.

Average cost per request hides the expensive tail and failed work.

## Remove Waste Before Changing Models

The first pass is mechanical:

1. **Deduplicate logical operations.** Use idempotency keys so a client retry or
   queue redelivery does not generate twice.
2. **Bound retries.** All attempts share an end-to-end deadline and retry budget.
3. **Limit output.** Set task-specific maximum output and a clear stopping format.
4. **Trim context.** Remove repeated logs, stale history, unused tool schemas, and
   irrelevant retrieval results.
5. **Stop dead workflows.** Cancel downstream model calls after validation or user
   abandonment makes the result unusable.
6. **Move deterministic work to code.** Parsing, arithmetic, permissions, and
   exact formatting do not need a generative call.

Measure each change against the same eval set. Shortening context can cut tokens
and still reduce correctness if the removed evidence mattered.

## Structure Prompts for Provider Caching

Provider prompt caches usually reward a stable prefix. Put durable instructions,
schemas, and shared examples before user-specific data. Keep volatile timestamps,
IDs, and conversation state later.

```text
stable policy and output schema
shared task examples
retrieved or tenant-specific context
current user input
```

Do not pad a prompt to reach a cache threshold. More input still has latency,
privacy, and quality cost. Instrument the provider's usage fields and compare:

- eligible input tokens;
- cache writes and reads;
- hit rate by prompt version;
- billed cost with and without the cache;
- retention and data-processing compatibility.

Provider rules, thresholds, retention, and prices change. Link the implementation
to current documentation and keep the financial model outside the article or
source code.

## Set Output Budgets per Task

Output can dominate cost. A global `maxTokens` is not a product requirement.
Define a ceiling and shape for each task:

| Task | Better contract |
|---|---|
| Classification | Enum only |
| Extraction | JSON Schema with required fields |
| Search summary | Fixed number of cited bullets |
| Agent planning | Bounded steps and tool budget |
| Long report | Section limits with continuation workflow |

Track requested versus used output. If answers regularly stop early, lower the
ceiling. If they hit the cap and fail validation, fix the task design rather than
silently raising every request.

## Route Stable Tasks, Not “Easy Prompts”

Model routing saves money only when eligibility is based on an evaluated task.
A cheap classifier deciding that an arbitrary user message “looks easy” can add
another model call and a new failure mode.

Start with explicit task names. For each candidate model, compare:

- task-contract pass rate;
- harmful failure rate;
- latency distribution;
- average attempts and fallback rate;
- cost per passed task;
- performance by language and important customer slice.

Use offline evals, then shadow traffic, then a reversible canary. Promote the
candidate only for the tasks it passed. The
[multi-provider routing guide](/blog/multi-provider-llm-architecture/) covers route
eligibility and failover.

Do not route around a provider's safety or data policy. A cheaper model that cannot
serve the required region or structured output is not eligible.

## Move Delay-Tolerant Work to Batch

Batch endpoints can have different pricing and completion windows. They fit work
that is already asynchronous:

- nightly evaluation runs;
- document enrichment;
- offline classification;
- embedding backfills;
- report generation with a clear deadline.

Batching changes operations. Jobs need stable item IDs, partial-failure handling,
expiry, retries, output validation, and reconciliation. Never assume output order
matches input order; join by the provider-supported custom identifier.

Compare the full job cost, including storage, polling or webhooks, failed items,
and missed deadlines. Check current provider documentation before relying on a
discount or completion window.

## Use Exact Caching Before Semantic Caching

Exact caching is easier to reason about. It works for deterministic inputs such as
an immutable document hash plus a prompt version.

A safe cache key may include:

```text
tenant + permissions + task + normalized input hash + source versions +
prompt version + model version + locale + policy version
```

Semantic caching reuses an answer for a merely similar request. Similar wording
does not guarantee equivalent intent, authorization, time, or source state. Avoid
it for legal, medical, financial, account, personalized, or rapidly changing
answers unless the equivalence rule is validated for that task.

If you use it:

- scope entries by tenant and permissions;
- set freshness from the source, not a universal TTL;
- store provenance and the exact input that produced the answer;
- re-run deterministic policy checks on every hit;
- invalidate on prompt, model, policy, or source changes;
- audit false hits with human-labeled examples;
- include embedding, vector store, and miss-path cost in the ledger.

A cache hit is a product decision, not automatically a success.

## Control Budgets at Three Levels

**Request budget:** input/output ceiling, deadline, attempts, tools, and fallback.

**Tenant budget:** daily or monthly spend, alerts, feature-specific limits, and a
declared degraded mode.

**System budget:** provider commitments, capacity, incident reserve, and financial
forecast.

Spend-limit enforcement may be eventually consistent. Keep application-side
limits for actions that must stop precisely, and decide what the user sees when a
budget is exhausted: smaller eligible route, queued processing, limited feature,
or no generation.

## Run Cost Changes Like Product Releases

For every optimization:

1. write the hypothesis and affected task;
2. freeze the eval set and baseline route;
3. estimate token, infrastructure, and review impact;
4. shadow when output behavior can change;
5. canary a small cohort;
6. watch quality, latency, attempts, and cost per success;
7. promote or roll back with a recorded reason.

Do not stack prompt trimming, a model swap, a new cache, and different retry logic
in one release. You will not know which change saved money or broke quality.

## Production Checklist

- [ ] Spend is attributed to task, route, attempt, tenant, and outcome.
- [ ] Estimated price tables are versioned and reconciled with invoices.
- [ ] Idempotency, bounded retries, cancellation, and output limits are in place.
- [ ] Context sections are measured and trimmed against evals.
- [ ] Cache economics use observed reads, writes, and billed tokens.
- [ ] Smaller models pass task-specific eval, shadow, and canary gates.
- [ ] Batch jobs handle identity, partial failure, expiry, and validation.
- [ ] Cache keys include authorization, freshness, and all behavior versions.
- [ ] Budget exhaustion has a safe, user-visible outcome.
- [ ] Dashboards report cost per successful task, not only total tokens.

## Primary References

- [OpenAI: Prompt caching](https://platform.openai.com/docs/guides/prompt-caching)
- [OpenAI: Batch API](https://platform.openai.com/docs/guides/batch)
- [Anthropic: Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
- [Anthropic: Message Batches](https://platform.claude.com/docs/en/build-with-claude/batch-processing)
- [Google Cloud: Vertex AI context cache](https://cloud.google.com/vertex-ai/generative-ai/docs/context-cache/context-cache-overview)

The best optimization is not the lowest bill. It is the lowest reproducible cost
for a task that still passes its contract.