LLM Cost Optimization: Measure Cost per Successful Task

“Cut LLM cost by 60%” is not an engineering target unless the baseline, traffic, and quality bar are stated. A cheaper request that fails twice, reaches a human, or damages conversion can cost more than the original.

Use this metric instead:

cost per successful task =
  (model + embeddings + reranking + retries + cache + review + infrastructure)
  / tasks that pass the product outcome and quality contract

The production LLM stack guide defines that task contract. This article focuses on the cost controls inside it.

Build a Cost Ledger First

Provider invoices aggregate spend. They rarely explain which product decision created it. Record one ledger row per logical task:

interface LlmCostEvent {
  taskId: string;
  taskType: string;
  tenantId: string;
  promptVersion: string;
  routeVersion: string;
  provider: string;
  model: string;
  attempt: number;
  inputTokens?: number;
  cachedInputTokens?: number;
  outputTokens?: number;
  estimatedCost: number;
  latencyMs: number;
  outcome: 'passed' | 'failed' | 'escalated' | 'abandoned';
}

Keep price tables versioned by provider, model, region, service tier, and date. Reconcile estimates with the invoice. Do not log raw prompts or personal data just to understand cost.

Break dashboards down by task and outcome:

• total and marginal spend;
• cost per passed task;
• tokens per section: policy, retrieval, history, tools, output;
• attempts and fallback share;
• cache read/write tokens and hit rate;
• quality pass, escalation, and abandonment rates;
• P50/P95 latency;
• spend by tenant with budget alerts.

Average cost per request hides the expensive tail and failed work.

Remove Waste Before Changing Models

The first pass is mechanical:

1. Deduplicate logical operations. Use idempotency keys so a client retry or queue redelivery does not generate twice.
2. Bound retries. All attempts share an end-to-end deadline and retry budget.
3. Limit output. Set task-specific maximum output and a clear stopping format.
4. Trim context. Remove repeated logs, stale history, unused tool schemas, and irrelevant retrieval results.
5. Stop dead workflows. Cancel downstream model calls after validation or user abandonment makes the result unusable.
6. Move deterministic work to code. Parsing, arithmetic, permissions, and exact formatting do not need a generative call.

Measure each change against the same eval set. Shortening context can cut tokens and still reduce correctness if the removed evidence mattered.

Structure Prompts for Provider Caching

Provider prompt caches usually reward a stable prefix. Put durable instructions, schemas, and shared examples before user-specific data. Keep volatile timestamps, IDs, and conversation state later.

stable policy and output schema
shared task examples
retrieved or tenant-specific context
current user input

Do not pad a prompt to reach a cache threshold. More input still has latency, privacy, and quality cost. Instrument the provider’s usage fields and compare:

• eligible input tokens;
• cache writes and reads;
• hit rate by prompt version;
• billed cost with and without the cache;
• retention and data-processing compatibility.

Provider rules, thresholds, retention, and prices change. Link the implementation to current documentation and keep the financial model outside the article or source code.

Set Output Budgets per Task

Output can dominate cost. A global maxTokens is not a product requirement. Define a ceiling and shape for each task:

Track requested versus used output. If answers regularly stop early, lower the ceiling. If they hit the cap and fail validation, fix the task design rather than silently raising every request.

Route Stable Tasks, Not “Easy Prompts”

Model routing saves money only when eligibility is based on an evaluated task. A cheap classifier deciding that an arbitrary user message “looks easy” can add another model call and a new failure mode.

Start with explicit task names. For each candidate model, compare:

• task-contract pass rate;
• harmful failure rate;
• latency distribution;
• average attempts and fallback rate;
• cost per passed task;
• performance by language and important customer slice.

Use offline evals, then shadow traffic, then a reversible canary. Promote the candidate only for the tasks it passed. The multi-provider routing guide covers route eligibility and failover.

Do not route around a provider’s safety or data policy. A cheaper model that cannot serve the required region or structured output is not eligible.

Move Delay-Tolerant Work to Batch

Batch endpoints can have different pricing and completion windows. They fit work that is already asynchronous:

• nightly evaluation runs;
• document enrichment;
• offline classification;
• embedding backfills;
• report generation with a clear deadline.

Batching changes operations. Jobs need stable item IDs, partial-failure handling, expiry, retries, output validation, and reconciliation. Never assume output order matches input order; join by the provider-supported custom identifier.

Compare the full job cost, including storage, polling or webhooks, failed items, and missed deadlines. Check current provider documentation before relying on a discount or completion window.

Use Exact Caching Before Semantic Caching

Exact caching is easier to reason about. It works for deterministic inputs such as an immutable document hash plus a prompt version.

A safe cache key may include:

tenant + permissions + task + normalized input hash + source versions +
prompt version + model version + locale + policy version

Semantic caching reuses an answer for a merely similar request. Similar wording does not guarantee equivalent intent, authorization, time, or source state. Avoid it for legal, medical, financial, account, personalized, or rapidly changing answers unless the equivalence rule is validated for that task.

If you use it:

• scope entries by tenant and permissions;
• set freshness from the source, not a universal TTL;
• store provenance and the exact input that produced the answer;
• re-run deterministic policy checks on every hit;
• invalidate on prompt, model, policy, or source changes;
• audit false hits with human-labeled examples;
• include embedding, vector store, and miss-path cost in the ledger.

A cache hit is a product decision, not automatically a success.

Control Budgets at Three Levels

Request budget: input/output ceiling, deadline, attempts, tools, and fallback.

Tenant budget: daily or monthly spend, alerts, feature-specific limits, and a declared degraded mode.

System budget: provider commitments, capacity, incident reserve, and financial forecast.

Spend-limit enforcement may be eventually consistent. Keep application-side limits for actions that must stop precisely, and decide what the user sees when a budget is exhausted: smaller eligible route, queued processing, limited feature, or no generation.

Run Cost Changes Like Product Releases

For every optimization:

1. write the hypothesis and affected task;
2. freeze the eval set and baseline route;
3. estimate token, infrastructure, and review impact;
4. shadow when output behavior can change;
5. canary a small cohort;
6. watch quality, latency, attempts, and cost per success;
7. promote or roll back with a recorded reason.

Do not stack prompt trimming, a model swap, a new cache, and different retry logic in one release. You will not know which change saved money or broke quality.

Production Checklist

• Spend is attributed to task, route, attempt, tenant, and outcome.
• Estimated price tables are versioned and reconciled with invoices.
• Idempotency, bounded retries, cancellation, and output limits are in place.
• Context sections are measured and trimmed against evals.
• Cache economics use observed reads, writes, and billed tokens.
• Smaller models pass task-specific eval, shadow, and canary gates.
• Batch jobs handle identity, partial failure, expiry, and validation.
• Cache keys include authorization, freshness, and all behavior versions.
• Budget exhaustion has a safe, user-visible outcome.
• Dashboards report cost per successful task, not only total tokens.

Primary References

• OpenAI: Prompt caching
• OpenAI: Batch API
• Anthropic: Prompt caching
• Anthropic: Message Batches
• Google Cloud: Vertex AI context cache

The best optimization is not the lowest bill. It is the lowest reproducible cost for a task that still passes its contract.