AI Cost Optimization: How to Cut LLM Spending by 60% Without Losing Quality

The average LLM startup spends $8,000–15,000 per month on API calls at 50,000 requests per day. Yet 40–60% of that goes toward repeated requests, excess tokens, and routing expensive models at tasks they’re massively overqualified for.

A 60% cost reduction is doable in 2–3 weeks. Without quality loss, without dropping flagship models on hard tasks. Five techniques, each with measurable results.

Anatomy of LLM costs: where the money goes

Before you cut anything, understand what you’re actually paying for. A typical breakdown for a production app with an AI chat and several pipelines:

Each technique below targets a specific category. The order follows ROI: the first one gives the most impact with the least effort.

Prompt Caching: saving on repeated prefixes

Prompt caching lets the provider cache the beginning of a prompt — system prompt, context, instructions — and skip reprocessing those tokens on every subsequent request that shares the same prefix.

Anthropic introduced prompt caching in August 2024. OpenAI shipped automatic caching in October 2024. Google Gemini has had context caching since mid-2024.

How it works. The provider hashes the first N tokens. If the next request starts with the same block, cached tokens aren’t billed at full price. Anthropic’s discount: 90% on cached tokens. OpenAI’s: 50%.

Anthropic: explicit caching

With Anthropic, caching is managed explicitly via the cache_control parameter in messages:

{
  "model": "claude-sonnet-4-6",
  "max_tokens": 1024,
  "system": [
    {
      "type": "text",
      "text": "You are an AI assistant for travel planning. Respond in a structured format, use markdown. Always include budget, travel time, safety rating for each location...",
      "cache_control": {"type": "ephemeral"}
    }
  ],
  "messages": [
    {"role": "user", "content": "Plan a 7-day itinerary in Bali"}
  ]
}

Minimum cacheable block size: 1,024 tokens for Claude Sonnet/Opus, 2,048 for Haiku. The cache lives for 5 minutes and is refreshed with each access.

OpenAI: automatic caching

With OpenAI, prompt caching works automatically for prompts longer than 1,024 tokens. No code changes needed. The longest common prefix between requests is cached.

Savings calculation

System prompt for an AI chat — 2,000 tokens. 10,000 requests per day. Model: Claude Sonnet 4.6.

Without caching:

• System prompt input tokens: 2,000 × 10,000 = 20M tokens/day
• Cost: 20M × ~$3.00/1M = ~$60/day

With caching (Anthropic, 90% discount on cached tokens):

• First request: ~$3.00/1M (full price)
• Cached: ~$0.30/1M
• Cache write: ~$3.75/1M (25% more than standard, only on first write)
• Cost: 20M × ~$0.30/1M = ~$6/day

Savings: ~$54/day, ~$1,620/month. And that’s just the system prompt. Cache few-shot examples and additional context, and the savings scale proportionally.

Pattern: maximizing cache hits

Prompt structure determines caching efficiency. The cacheable block goes first; variable parts go last.

┌─────────────────────────────┐
│  System prompt (cached)     │  ← 2000+ tokens, same for everyone
├─────────────────────────────┤
│  Few-shot examples          │  ← 500–1000 tokens, same for everyone
│  (cached)                   │
├─────────────────────────────┤
│  User context               │  ← Unique per request
├─────────────────────────────┤
│  User message               │  ← Unique per request
└─────────────────────────────┘

Common mistake: putting the user’s name at the top of the system prompt. Every new user breaks the cache. Fix: put the name after the cached block, or move it into the user message.

Smart Routing: the right model for each task

Not every request needs a flagship model. Classifying a ticket, generating a headline, pulling an email from text — GPT-5.4-mini or Gemini 3.1 Flash handles these just as well as GPT-5.4 at 10–30x lower cost.

Model pricing table (April 2026)

Prices are approximate as of publication and may change. Check provider websites for current rates.

The gap between Gemini Flash and Claude Opus is 200x on input, 250x on output. Not a rounding error — orders of magnitude.

Task classifier

Routing starts with classification. Two approaches: rule-based (by task type) or LLM-based (a cheap model decides complexity).

Rule-based routing — simpler, more predictable, sufficient for most cases:

ROUTING_TABLE = {
    # task_type: model
    "chat_title_generation": "gemini/gemini-2.0-flash",
    "ticket_classification": "gpt-5.4-mini",
    "email_extraction": "gpt-5.4-mini",
    "sentiment_analysis": "gemini/gemini-2.0-flash",
    "translation": "deepseek/deepseek-chat",
    "code_review": "anthropic/claude-sonnet-4-6",
    "complex_reasoning": "anthropic/claude-sonnet-4-6",
    "trip_planning": "anthropic/claude-sonnet-4-6",
}

def route_request(task_type: str, fallback: str = "gpt-5.4-mini") -> str:
    return ROUTING_TABLE.get(task_type, fallback)

LLM-based router — for cases where task complexity is unpredictable. A cheap model evaluates the incoming request and routes it to the appropriate model:

ROUTER_PROMPT = """Assess the complexity of the user's task on a scale of 1–3:
1 - simple (classification, extraction, formatting)
2 - medium (summarization, translation, answering a question)
3 - complex (reasoning, analysis, plan generation)

Reply with ONLY the number: 1, 2, or 3."""

COMPLEXITY_TO_MODEL = {
    1: "gpt-5.4-mini",              # ~$0.15/~$0.60
    2: "deepseek/deepseek-chat",   # ~$0.14/~$0.28
    3: "anthropic/claude-sonnet-4-6", # ~$3/~$15
}

The router itself costs ~50 input tokens + ~1 output token via GPT-5.4-mini — $0.000008 per classification. At 10,000 requests/day that’s $0.08/day. Routing savings: tens of dollars per day.

Real distribution

In practice, 60–70% of requests in a typical AI app are “simple,” 20–25% are “medium,” and only 10–15% actually need a flagship model.

Savings: ~$71.50/day, ~$2,145/month.

More on setting up multi-provider routing through LiteLLM in the multi-provider architecture article.

Batch API: deferred tasks at half price

OpenAI, Anthropic, and Google all offer Batch APIs — submit a batch, get results within 24 hours. Discount: 50% off standard pricing.

Not everything needs a 200ms response. Weekly reports, bulk content classification, data enrichment, processing uploaded documents — all of this can wait.

Which tasks work for Batch API

OpenAI Batch API

import json
from openai import OpenAI

client = OpenAI()

# 1. Prepare JSONL file with requests
requests = []
for i, item in enumerate(items_to_classify):
    requests.append({
        "custom_id": f"item-{i}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-5.4-mini",
            "messages": [
                {"role": "system", "content": "Classify the product into categories: electronics, clothing, food, other. Reply with one word."},
                {"role": "user", "content": item["description"]}
            ],
            "max_tokens": 10
        }
    })

# Write to file
with open("batch_input.jsonl", "w") as f:
    for req in requests:
        f.write(json.dumps(req) + "\n")

# 2. Upload file
batch_file = client.files.create(
    file=open("batch_input.jsonl", "rb"),
    purpose="batch"
)

# 3. Create batch
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

# 4. Check status (later)
status = client.batches.retrieve(batch.id)
# status.status: "validating" → "in_progress" → "completed"

Anthropic Message Batches

Anthropic has an equivalent mechanism — Message Batches API. 50% discount, results within 24 hours.

import anthropic

client = anthropic.Anthropic()

batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"item-{i}",
            "params": {
                "model": "claude-sonnet-4-6",
                "max_tokens": 100,
                "messages": [
                    {"role": "user", "content": f"Classify: {item['description']}"}
                ]
            }
        }
        for i, item in enumerate(items)
    ]
)

Savings calculation

1,000 documents per day for classification. Average request: 500 input + 20 output tokens. Model: GPT-5.4-mini.

Synchronous:

• Input: 500K × ~$0.15/1M = ~$0.075
• Output: 20K × ~$0.60/1M = ~$0.012
• Total: ~$0.087/day

Batch API (50% discount):

• Total: ~$0.044/day

On GPT-5.4-mini the absolute numbers are modest. But run the same math for Claude Sonnet 4.6 at 10,000 requests:

Synchronous: ~$45/day → Batch: ~$22.50/day. Savings: ~$675/month.

Semantic Caching: identical questions don’t need to be processed twice

Prompt caching works at the provider level (repeated prefix). Semantic caching works at the application level: if a user asks something similar to a question you’ve already processed, the answer comes from local cache — no LLM call.

“What’s the weather in Bali in December?” and “Bali weather in December” are the same question. There’s no reason to pay twice.

How semantic cache works

User request
    │
    ▼
Generate embedding (text-embedding-3-small: $0.02/1M tokens)
    │
    ▼
Search in vector store (Qdrant / Redis / pgvector)
    │
    ├── similarity > 0.95 → return cached response (0 LLM tokens)
    │
    └── similarity < 0.95 → call LLM → save response + embedding to cache

Implementation with Redis + OpenAI embeddings

import hashlib
import json
import numpy as np
import redis
from openai import OpenAI

client = OpenAI()
cache = redis.Redis(host="localhost", port=6379, db=0)

SIMILARITY_THRESHOLD = 0.95
CACHE_TTL = 3600  # 1 hour

def get_embedding(text: str) -> list[float]:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

def cosine_similarity(a: list[float], b: list[float]) -> float:
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

def cached_completion(messages: list[dict], model: str = "gpt-5.4") -> str:
    user_message = messages[-1]["content"]
    query_embedding = get_embedding(user_message)

    # Search cache
    cached_keys = cache.keys("sem_cache:*")
    for key in cached_keys:
        cached_data = json.loads(cache.get(key))
        similarity = cosine_similarity(query_embedding, cached_data["embedding"])
        if similarity >= SIMILARITY_THRESHOLD:
            return cached_data["response"]

    # Cache miss: call LLM
    response = client.chat.completions.create(
        model=model,
        messages=messages
    )
    result = response.choices[0].message.content

    # Save to cache
    cache_key = f"sem_cache:{hashlib.md5(user_message.encode()).hexdigest()}"
    cache.setex(
        cache_key,
        CACHE_TTL,
        json.dumps({"embedding": query_embedding, "response": result})
    )

    return result

For production, replace linear search over Redis keys with a proper vector database — Qdrant, Pinecone, or pgvector. At 10,000+ cache entries, linear search will become the bottleneck.

GPTCache: ready-made solution

GPTCache by Zilliz — an open-source library for semantic caching. Supports multiple embedding models, vector stores, and eviction strategies.

from gptcache import cache
from gptcache.adapter import openai
from gptcache.embedding import OpenAI as OpenAIEmbedding
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation import SearchDistanceEvaluation

# Initialization
embedding = OpenAIEmbedding()
cache_base = CacheBase("sqlite")
vector_base = VectorBase("faiss", dimension=embedding.dimension)

cache.init(
    embedding_func=embedding.to_embeddings,
    data_manager=get_data_manager(cache_base, vector_base),
    similarity_evaluation=SearchDistanceEvaluation()
)

# Usage — interface identical to openai
response = openai.ChatCompletion.create(
    model="gpt-5.4",
    messages=[{"role": "user", "content": "Bali weather in December"}]
)

Savings calculation

Assumptions: 10,000 requests per day, 30% cache hit rate (conservative for FAQ-like scenarios), model GPT-5.4 (approximate prices used).

Without semantic cache:

• 10,000 × (500 input + 200 output tokens)
• Cost: 5M × ~$2/1M + 2M × ~$8/1M = ~$26/day

With semantic cache (30% hits):

• 7,000 LLM requests: ~$18.20
• 10,000 embeddings: 5M × ~$0.02/1M = ~$0.10
• Total: ~$18.30/day

Savings: ~$7.70/day, ~$230/month. At 50% cache hit rate — realistic for support bots — that climbs to ~$390/month.

Model Downsizing: systematic replacement of expensive models

The most obvious technique, and the riskiest if you skip the process. Swapping Claude Sonnet 4.6 for GPT-5.4-mini on a specific task isn’t flipping a switch. It requires an eval framework.

Downsizing process

1. Collect an eval dataset. 50–100 real examples from production logs for each task. Input data + expected responses (or quality criteria).

2. Run a baseline. Current model on the eval dataset. Record metrics: accuracy, format compliance, latency, cost.

3. Run the candidate. Cheaper model on the same dataset. Same metrics.

4. Compare. If the cheaper model delivers >95% of the quality of the expensive one — safe to switch.

5. A/B test in production. 10% of traffic to the new model, monitoring via Langfuse or equivalent.

Eval framework

import json
from dataclasses import dataclass

@dataclass
class EvalResult:
    model: str
    task: str
    accuracy: float
    avg_latency_ms: float
    cost_per_1k_requests: float
    format_compliance: float  # % of responses in the expected format

def run_eval(model: str, task: str, dataset: list[dict]) -> EvalResult:
    results = []
    total_cost = 0
    total_latency = 0
    correct = 0
    format_ok = 0

    for example in dataset:
        start = time.time()
        response = call_llm(model=model, messages=example["messages"])
        latency = (time.time() - start) * 1000

        total_latency += latency
        total_cost += response.usage.total_cost

        if evaluate_correctness(response.content, example["expected"]):
            correct += 1
        if validate_format(response.content, example.get("format_schema")):
            format_ok += 1

        results.append(response)

    n = len(dataset)
    return EvalResult(
        model=model,
        task=task,
        accuracy=correct / n,
        avg_latency_ms=total_latency / n,
        cost_per_1k_requests=(total_cost / n) * 1000,
        format_compliance=format_ok / n,
    )

# Comparison
baseline = run_eval("anthropic/claude-sonnet-4-6", "ticket_classification", dataset)
candidate = run_eval("gpt-5.4-mini", "ticket_classification", dataset)

print(f"Accuracy: {baseline.accuracy:.1%} → {candidate.accuracy:.1%}")
print(f"Cost/1K:  ${baseline.cost_per_1k_requests:.2f} → ${candidate.cost_per_1k_requests:.2f}")
print(f"Latency:  {baseline.avg_latency_ms:.0f}ms → {candidate.avg_latency_ms:.0f}ms")

Typical downsizing results

That last row is why eval isn’t optional. Cheap models fall apart on complex reasoning. Skip the eval, and you’ll degrade the product while thinking you’re saving money.

Combined strategy: summary table

Each technique alone cuts 10–50% in its area. Combined, the effects stack.

Calculation for an application with 10,000 requests/day, average request 500 input + 200 output tokens, base model Claude Sonnet 4.6:

Baseline: ~$2,700/month (everything on Claude Sonnet 4.6, no optimizations).

After optimization: ~$900–1,100/month.

Total savings: 59–67%. Implementation order: prompt caching (instant effect, zero risk) → smart routing (biggest absolute savings, needs task classification) → batch API (migrate deferred tasks) → semantic caching → model downsizing (last, because it requires eval).

Cost monitoring: without observability, optimization is blind

Optimizing without monitoring is guesswork. Three metrics to track daily:

Cost per request — average cost of one LLM call, broken down by task and model. A sudden spike usually means the cache broke, routing started sending to the expensive model, or someone padded the prompt.

Cache hit rate — for both prompt caching and semantic caching. If it drops below a threshold (say, <20% for semantic cache), something changed: request patterns shifted, TTL is too short, or the similarity threshold is too tight.

Quality score — accuracy and relevance on the eval dataset. Run it after any model swap or downsizing to confirm you haven’t quietly degraded the product.

Langfuse stores every trace: model, tokens, cost, latency. A cost-by-model dashboard takes minutes to build. LiteLLM integrates with Langfuse natively — every call through the proxy gets logged automatically.

# litellm_config.yaml
litellm_settings:
  success_callback: ["langfuse"]

environment_variables:
  LANGFUSE_PUBLIC_KEY: "pk-..."
  LANGFUSE_SECRET_KEY: "sk-..."
  LANGFUSE_HOST: "https://cloud.langfuse.com"

What not to do

Optimize before you’ve launched. Premature optimization applies to LLM costs as much as anything else. Ship first, get real usage data, then cut based on facts.

Trade quality for savings. If downsizing drops accuracy from 95% to 80%, that’s not optimization — that’s breaking the product. Users churn, and the $500/month you saved won’t come close to covering it.

Overlook hidden costs. Semantic caching needs a vector database. An eval framework needs time to label the dataset. Factor those into the ROI before counting the win.

Forget that models get cheaper. GPT-5.4 launched at $5/$15, now it’s ~$2/$8. Gemini Flash started on a free tier, now ~$0.075/$0.30. Check provider pricing every 2–3 months — the math changes.

Conclusion

Five techniques applied in order cut LLM costs by 60% or more. Prompt caching and batch API need no architectural changes and pay off on day one. Smart routing and model downsizing take more work, but produce the largest absolute numbers.

One rule holds across all of them: every change needs to be measurable. Run eval before downsizing. Watch metrics after any change. Keep a cost-per-request dashboard in Langfuse or equivalent. Without that visibility, you’re not optimizing — you’re guessing, and savings quietly turn into degradation.

Need help with LLM cost optimization? I help startups build AI products and automate processes — belov.works.