# What Are Tokens in LLMs? How to Count Them and Cut Costs

> What a token is in an LLM, how many tokens a word or page takes in English and Russian, how to count them before and after a call, and what they cost.
> Author: Roman Belov · Published: 2026-10-17 · Source: https://futurecraft.pro/blog/tokens-explained/

A token is a piece of text that a tokenizer splits a language model's input into: a word, part of a word, a character, or a punctuation mark. Each piece maps to a numeric ID in the tokenizer's vocabulary, and the model works with the sequence of these numbers, not with letters. Context windows are measured in tokens, and API bills are counted in tokens.

In practice, three questions follow from this definition: how many tokens will my text take, how much will it cost, and why doesn't the bill match my estimate. The answers depend on the model. Each one has its own tokenizer, and the same paragraph can turn into 330 tokens on one model and 520 on another.

This article covers how tokens work, how many there are in a word, a page, and a million, how input, output, and thinking tokens differ, how to count them before and after a request, and what they cost on November 8, 2026. The original material is the author's measurement from 2026-11-08: the same texts in Russian and English run through eight tokenizers (cloud models via OpenRouter, local ones via Ollama 0.40.14). Everything else comes from vendor documentation, linked in the text.

## What is a token in an LLM

A model doesn't read letters and doesn't know words. On the way in, the tokenizer turns a string into a list of integers. On the way out, the model produces the next integer, and the tokenizer turns it back into text. A tokenizer vocabulary holds tens to hundreds of thousands of pieces: frequent words as a whole, rare words in parts, and single bytes for everything else.

### How text is split into tokens

Most current tokenizers are built on BPE (byte pair encoding) and its variants. Sennrich, Haddow, and Birch described the approach for neural machine translation in 2015 ([arXiv:1508.07909](https://arxiv.org/abs/1508.07909)): the vocabulary is built from the most frequent symbol pairs, so common words get their own token and rare words are assembled from pieces. OpenAI's [tiktoken](https://github.com/openai/tiktoken) library works the same way at the byte level.

Here is how one phrase splits in two OpenAI encodings (tiktoken 0.14.0 output), with two Russian rows for comparison:

| Text | o200k_base | cl100k_base |
|---|---|---|
| `A token is a piece of text.` | 8 tokens: `A` ` token` ` is` ` a` ` piece` ` of` ` text` `.` | 8 tokens |
| `Токен это фрагмент текста.` (Russian) | 9 tokens: `Т` `ок` `ен` ` это` ` ф` `раг` `мент` ` текста` `.` | 11 tokens |
| `tokenization` | 2 tokens: `token` `ization` | 2 tokens |
| `нейросети` (Russian, "neural networks") | 3 tokens: `ней` `рос` `ети` | 5 tokens: `н` `ей` `рос` `ет` `и` |

Almost every English word here is a single token, with the leading space included. The Russian word for "token" broke into three pieces. In the older cl100k_base encoding one Cyrillic letter was even split across two tokens at the byte level: Cyrillic takes 2 bytes per letter in UTF-8, and BPE sometimes cuts in the middle of a character.

To see how your own text splits:

```python
import tiktoken

enc = tiktoken.get_encoding("o200k_base")   # GPT-4o, GPT-5.x
text = "A token is a piece of text."
ids = enc.encode(text)
print(len(ids), [enc.decode([i]) for i in ids])
assert enc.decode(ids) == text
```

### LLM tokens vs crypto tokens

These are different things with the same name. In crypto, a token is a digital asset on a blockchain, and "asset tokenization" means issuing such assets. In language models, a token is a unit of text, and tokenization is splitting text into those units.

## How many tokens in a word, a page, and a million

There is no universal ratio, but vendors give rules of thumb. All of them refer to English text or to text with no language specified.

### Vendor rules of thumb

- **OpenAI (tiktoken README):** "On average, in practice, each token corresponds to about 4 bytes." These are bytes, not characters. In English that is about 4 characters. A Cyrillic letter takes 2 bytes, so the same bytes hold half as many characters.
- **Google:** for Gemini models, "a token is equivalent to about 4 characters. 100 tokens is equal to about 60-80 English words" ([docs](https://aistudio.google.com/docs/tokens)).
- **Anthropic:** "1M tokens is roughly 555k words or 2.5M Unicode characters on the current tokenizer (introduced with Claude Opus 4.7); models before it fit about 750k words" ([models overview](https://platform.claude.com/docs/en/models/overview)). Anthropic says the new tokenizer "produces approximately 30% more tokens for the same text."

### Russian vs English: measured on eight tokenizers

None of these vendors publishes a ratio for non-English text against English. To get one, the author measured it on Russian, a language with a non-Latin script.

There were three texts of 1,100-1,400 characters each: a technical paragraph, an everyday story, and Python code with comments. Each was written in Russian and translated into natural English, not word for word; in the code only comments and strings were translated. Cloud models were counted through OpenRouter as `prompt_tokens(text) - prompt_tokens("x") + 1` to subtract the chat wrapper, each run twice. tiktoken ran offline, Qwen3 and Gemma 4 through Ollama with `raw: true`.

| Tokenizer | Technical RU/EN | Everyday RU/EN | Code RU/EN | RU chars per token (tech.) | EN chars per token (tech.) |
|---|---|---|---|---|---|
| Claude Haiku 5.5 | 520 / 405 = 1.28 | 505 / 423 = 1.19 | 437 / 396 = 1.10 | 2.58 | 3.57 |
| GPT-6 Luna (native count) | 343 / 276 = 1.24 | 367 / 298 = 1.23 | 277 / 253 = 1.09 | 3.92 | 5.23 |
| tiktoken o200k_base | 343 / 276 = 1.24 | 367 / 298 = 1.23 | 277 / 253 = 1.09 | 3.92 | 5.23 |
| tiktoken cl100k_base (GPT-4) | 582 / 277 = 2.10 | 610 / 299 = 2.04 | 325 / 252 = 1.29 | 2.31 | 5.21 |
| Gemini 3.8 Flash | 328 / 277 = 1.18 | 353 / 297 = 1.19 | 315 / 297 = 1.06 | 4.10 | 5.21 |
| Gemma 4 e4b (Ollama) | 329 / 278 = 1.18 | 354 / 298 = 1.19 | 316 / 298 = 1.06 | 4.09 | 5.19 |
| DeepSeek V4.1 Flash | 389 / 276 = 1.41 | 433 / 301 = 1.44 | 302 / 266 = 1.14 | 3.46 | 5.23 |
| Qwen3 8B (Ollama) | 439 / 277 = 1.58 | 470 / 299 = 1.57 | 294 / 255 = 1.15 | 3.06 | 5.21 |

*Author's measurement, 2026-11-08: cloud models via OpenRouter, local models via Ollama 0.40.14, tiktoken 0.14.0.*

### What the measurement shows

- The Russian text took more tokens than the English text with the same meaning on all eight tokenizers. On prose the gap ranged from 1.18x (Gemini, Gemma) to 1.58x (Qwen3), and was about 2x on the older cl100k_base.
- On code the gap was smaller, 1.06-1.15x: the code and identifiers are identical, only the comments were translated.
- GPT-6 Luna returned exactly the same counts as tiktoken o200k_base on all six texts. That's a match on six texts, not confirmation from OpenAI that GPT-6 uses o200k_base: OpenAI doesn't publish the GPT-6 encoding.
- Gemini 3.8 Flash and Gemma 4 gave almost identical numbers (0-1 token apart). That looks like a shared tokenizer, but it isn't officially confirmed.
- Claude Haiku 5.5 uses the most tokens in absolute terms on English too: 3.57 characters per token against about 5.2 for the rest. This matches Anthropic's note about the new tokenizer. Token count alone doesn't give you the cost, because the price per token differs between models.

With three texts, one per genre, this doesn't support a rule like "non-English always costs N times more." A study on Ukrainian, another Cyrillic language ([arXiv:2608.21384](https://arxiv.org/abs/2608.21384), July 2026), reports "68-121% token overhead" on modern tokenizers compared with English, and 220% on cl100k. That's a different language and a different method, so its figures don't transfer to Russian or to other languages.

### Tokens per page and what "a million tokens" means

Converted to a 1,800-character page (spaces included) and to a million tokens, the measurement looks like this. Ranges span the technical and everyday texts; code is excluded.

| Tokenizer | Tokens per word EN / RU | Tokens per page EN | Tokens per page RU | English words in 1M tokens |
|---|---|---|---|---|
| Claude Haiku 5.5 | 1.6-1.7 / 2.7-2.9 | 505-565 | 700-750 | 600-625K |
| GPT-6 Luna / o200k_base | 1.13-1.14 / 1.9-2.0 | 345-400 | 460-545 | about 880K |
| Gemini 3.8 Flash | 1.12-1.14 / 1.8-1.9 | 345-395 | 440-525 | about 880K |
| DeepSeek V4.1 Flash | 1.14 / 2.1-2.3 | 345-400 | 520-640 | about 880K |
| Qwen3 8B | 1.13-1.14 / 2.4-2.5 | 345-400 | 590-700 | about 880K |
| cl100k_base | 1.13-1.14 / 3.2 | 345-400 | 780-905 | about 880K |

On English, every tokenizer except Claude's landed at about 1.12-1.14 tokens per word. Claude Haiku 5.5 came to 600,000-625,000 English words per million tokens, the same order as Anthropic's estimate of 555,000.

## Input, output, and thinking tokens

The bill for a request is made of several kinds of tokens, and they are priced differently.

### Why output costs more and where thinking goes

The model processes input tokens (prompt, history, files) in a single pass but generates output tokens one at a time, and providers charge more for them. On November 8, 2026, output costs 5x input for all current Claude models and for the GPT-6 line.

Thinking tokens are output tokens too. At Anthropic, thinking is included in `output_tokens` and billed as output, and the `usage.output_tokens_details.thinking_tokens` field shows how many of the billed output tokens went to thinking ([docs](https://platform.claude.com/docs/en/build-with-claude/extended-thinking)). Opus 5.5, Sonnet 5.5, Haiku 5.5, and Fable 5.1 use adaptive thinking, with depth set by the effort parameter; a manual `budget_tokens` on 4.7 and later models returns a 400 error.

At OpenAI, reasoning tokens aren't visible in the response but are billed as output and reported in `usage.output_tokens_details.reasoning_tokens` ([docs](https://developers.openai.com/api/docs/guides/reasoning)). The `max_output_tokens` limit covers both reasoning and the visible answer. If the limit runs out during reasoning, the response comes back with status `incomplete` and reason `max_output_tokens`: you pay for input and reasoning and get no visible text. Gemini's output price also includes thinking tokens.

### Images, PDFs, and tools

Images become tokens too. Claude cuts an image into 28x28-pixel patches, each patch is one visual token, and the cost is `⌈width/28⌉ × ⌈height/28⌉` ([vision](https://platform.claude.com/docs/en/build-with-claude/vision)). On 4.7 and later models the cap is 4,784 tokens (long edge up to 2,576 px); older models stop at 1,568. Examples from the docs for current models: 1000x1000 is 1,296 tokens, 1920x1080 is 2,691, and 3840x2160 after downscaling is 4,784. Gemini charges 258 tokens for an image up to 384 px on both sides, and tiles larger images into 768x768 tiles at 258 tokens each.

Tools consume input even when the model doesn't call them. Names, descriptions, and JSON schemas from the `tools` parameter count as input tokens. On top of that, Anthropic adds a tool use system prompt: 286 tokens for Opus 5.5, Sonnet 5.5, and Haiku 5.5 in `auto` mode ([pricing](https://platform.claude.com/docs/en/about-claude/pricing)). In the token counting docs example, a request with one `get_weather` tool and a short user message counts as 403 input tokens.

Anthropic counts PDFs with the same `count_tokens`, but only when the document is passed as base64: the counting endpoint doesn't accept `url` and `file` sources.

### Cached tokens

If the start of a prompt repeats across requests, providers cache it and charge less to read it again. For Claude, a cache read costs 0.1x the input price; for Opus 5.5 and Sonnet 5.5 it is 0.05x, for Fable 5.1 0.025x. Writing to the 5-minute cache costs 1.25x, to the 1-hour cache 2x. In Claude's `usage`, input splits into three fields: `cache_read_input_tokens` (read from cache), `cache_creation_input_tokens` (written to cache) and `input_tokens` (only what comes after the last cache breakpoint). Total input is the sum of the three. At OpenAI and OpenRouter, the cached part is in `prompt_tokens_details.cached_tokens`.

## How to count tokens

You can count before a request, to estimate cost and fit the context window, and after it, to see what you were charged. Only the second matches the bill.

### Before the request: tiktoken, count_tokens, and Gemini

| Method | Models | Cost | What it counts |
|---|---|---|---|
| `tiktoken` (o200k_base) | GPT-4o, GPT-5.x; estimate only for GPT-6 | Free, offline | Text only |
| Anthropic `POST /v1/messages/count_tokens` | All active Claude models | Free, separate RPM limit | system, messages, tools, base64 images and PDFs |
| Gemini `client.models.count_tokens` | Gemini models | Not checked for this article | contents |

tiktoken has a catch: the library's model table (main branch on November 8, 2026) has no entries for `gpt-6`. The prefixes `gpt-5`, `gpt-4.1`, `gpt-4o`, `o1-` and `o3-` map to `o200k_base`, while `tiktoken.encoding_for_model("gpt-6-luna")` fails with `KeyError`. For GPT-6, set the encoding explicitly with `get_encoding("o200k_base")` and treat the result as an estimate.

Anthropic provides counting as a separate endpoint that takes the same body as the Messages API ([token counting](https://platform.claude.com/docs/en/build-with-claude/token-counting)). It's free, and its limits of 5,000, 10,000, and 20,000 RPM on the Start, Build, and Scale tiers are separate from generation limits. The [Claude API guide](https://futurecraft.pro/blog/claude-api-developer-guide/) covers how Claude API calls work in general.

```python
import anthropic

client = anthropic.Anthropic()
r = client.messages.count_tokens(
    model="claude-opus-5-5",
    system="You are a technical editor.",
    messages=[{"role": "user", "content": "What is a token?"}],
)
print(r.input_tokens)   # API response: {"input_tokens": N}
```

Tokens are counted with the tokenizer of the model passed in `model`. The endpoint doesn't accept server tools (web search, code execution, and others, except the advisor tool), the MCP connector, or images by `url`. Caching isn't applied during counting.

For Gemini, counting goes through the google-genai SDK:

```python
from google import genai

client = genai.Client()
total = client.models.count_tokens(model="gemini-3.8-flash", contents="What is a token?")
print(total.total_tokens)
```

### After the request: the usage field

Every API response carries the actual consumption. These are the numbers that end up on the bill.

| API | Input | Output | Thinking | Cache |
|---|---|---|---|---|
| Anthropic Messages | `input_tokens` | `output_tokens` | `output_tokens_details.thinking_tokens` | `cache_creation_input_tokens`, `cache_read_input_tokens` |
| OpenAI / OpenRouter Chat Completions | `prompt_tokens` | `completion_tokens` | `completion_tokens_details.reasoning_tokens` | `prompt_tokens_details.cached_tokens` |
| OpenAI Responses API | `input_tokens` | `output_tokens` | `output_tokens_details.reasoning_tokens` | `input_tokens_details.cached_tokens` |
| Ollama `/api/generate`, `/api/chat` | `prompt_eval_count` | `eval_count` | - | - |

OpenRouter is the easiest way to compare models. Usage comes back in every response automatically (the `usage.include` and `stream_options.include_usage` parameters are deprecated and do nothing), tokens are counted with the model's native tokenizer, and the `cost` field shows the charged amount in dollars ([usage accounting](https://openrouter.ai/docs/use-cases/usage-accounting)). The measurement above was done this way. The [OpenRouter guide](https://futurecraft.pro/blog/openrouter-guide/) covers the setup.

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://openrouter.ai/api/v1",
                api_key=os.environ["OPENROUTER_API_KEY"])
resp = client.chat.completions.create(
    model="openai/gpt-6-luna", max_tokens=16,
    messages=[{"role": "user", "content": "What is a token?"}],
)
print(resp.usage.model_dump())   # prompt_tokens, completion_tokens, cost, ...
```

Local Ollama returns the counters in its final response. Without `raw: true`, Ollama wraps the prompt in the model's chat template, and the template's special tokens are counted in `prompt_eval_count`. To count clean text, the measurement used `raw: true` and subtracted the count of a one-word prompt. Other endpoints and parameters are covered in the [Ollama guide](https://futurecraft.pro/blog/ollama-guide/).

```bash
curl -s http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b", "prompt": "What is a token?",
  "raw": true, "stream": false, "options": {"num_predict": 1}
}' | jq '{prompt_eval_count, eval_count}'
```

### Why estimates don't match the bill

A gap between the estimate and the bill usually comes down to one of four causes.

1. **Different tokenizers.** The same technical paragraph in Russian took 328 tokens on Gemini 3.8 Flash and 520 on Claude Haiku 5.5. A tiktoken count doesn't carry over to Claude or Qwen.
2. **A new model generation.** According to Anthropic, the tokenizer in Claude 4.7 and later produces about 30% more tokens for the same text than Sonnet 4.6 and older. Counts taken on an older model need to be redone after migration.
3. **Request wrapping.** The chat template, the tool use system prompt, and role tokens are added to your text. That's why the measurement subtracted the count for a one-character request.
4. **An estimate is an estimate.** Anthropic states that `count_tokens` may differ from the actual input by a small amount, and that tokens added for system optimizations aren't billed.

## How much tokens cost

Prices are published per million tokens (MTok) and they change, so the table is a snapshot on a specific date.

### Prices per 1M tokens on November 8, 2026

| Model | Input | Output | Cache read |
|---|---|---|---|
| Claude Fable 5.1 | $10 | $50 | $0.25 |
| Claude Opus 5.5 | $4 | $20 | $0.20 |
| Claude Sonnet 5.5 | $2 | $10 | $0.10 |
| Claude Haiku 5.5 (prompt up to 100K) | $0.10 | $0.50 | $0.01 |
| Claude Haiku 5.5 (prompt over 100K) | $0.50 | $2.50 | $0.05 |
| GPT-6 Astra | $10 | $50 | $1 |
| GPT-6.1 Sol | $2 | $10 | $0.10 |
| GPT-6.1 Terra | $1 | $5 | $0.05 |
| GPT-6 Luna | $0.10 | $0.50 | $0.01 |
| Gemini 3.8 Flash | $0.75 | $3.75 | - |
| Gemini 3.5 Flash-Lite | $0.30 | $2.50 | - |
| DeepSeek V4.1 Flash | $0.15 / $0.30 | $0.60 / $1.20 | $0.003 / $0.006 |
| DeepSeek V4 Pro | $0.66 / $1.32 | $1.98 / $3.96 | $0.022 / $0.044 |

*Sources: [Anthropic](https://platform.claude.com/docs/en/about-claude/pricing), [OpenAI](https://developers.openai.com/api/docs/pricing), [Google](https://aistudio.google.com/docs/pricing), [DeepSeek](https://api-docs.deepseek.com/quick_start/pricing/), snapshot 2026-11-08.*

Notes on the table:

- For DeepSeek, the first figure is off-peak and the second is peak: 01:00-04:00 and 06:00-10:00 UTC on weekdays, except Chinese public holidays.
- Gemini 3.8 Flash goes up to $1.50 input and $7.50 output from January 1, 2027.
- Haiku 5.5 is priced by the prompt length of each request: if input including cache exceeds 100,000 tokens, the whole request is billed at the higher rate.
- The Batch API at Anthropic, OpenAI, and Google costs half the standard rate.

For choosing between Opus and Sonnet on price and task, see the [Claude Opus vs Sonnet comparison](https://futurecraft.pro/blog/claude-opus-vs-sonnet/).

### What a page of text costs: English vs Russian

The calculation covers input only: a 1,800-character page, token counts from the measurement (range between the technical and everyday texts), prices from the table above. For scale, it is a thousand pages, each sent as a separate request.

| Model | Tokens per page EN / RU | 1,000 pages EN | 1,000 pages RU |
|---|---|---|---|
| Claude Haiku 5.5 | 505-565 / 700-750 | $0.050-0.056 | $0.070-0.075 |
| Claude Sonnet 5.5 | 505-565 / 700-750 * | $1.01-1.13 | $1.39-1.50 |
| Claude Opus 5.5 | 505-565 / 700-750 * | $2.02-2.25 | $2.78-3.00 |
| Claude Fable 5.1 | 505-565 / 700-750 * | $5.05-5.63 | $6.96-7.49 |
| GPT-6 Luna | 345-400 / 460-545 | $0.034-0.040 | $0.046-0.054 |
| Gemini 3.8 Flash | 345-395 / 440-525 | $0.26-0.30 | $0.33-0.39 |
| DeepSeek V4.1 Flash (off-peak) | 345-400 / 520-640 | $0.052-0.060 | $0.078-0.096 |

\* Only Haiku 5.5 was measured. According to Anthropic, all Claude 4.7 and later models, including Opus 5.5, Sonnet 5.5, and Fable 5.1, use the tokenizer introduced with Claude Opus 4.7, so the token counts are taken from the Haiku measurement. This is a calculation, not a measurement.

Claude Haiku 5.5 and GPT-6 Luna have the same price per input token ($0.10 per million), yet Haiku uses about 1.4-1.5x more tokens on the same page, in both English and Russian. Comparing models by price per million without accounting for the tokenizer gives the wrong answer. And this is input only: an answer of the same length at the 5x output rate costs five times as much as the input.

Embeddings are billed by tokens as well. The [embeddings explainer](https://futurecraft.pro/blog/embeddings-explained/) covers what they are and how they work.

## Subscription limits: how many tokens in Claude Pro and ChatGPT Plus

For subscriptions, "how many tokens" has no exact answer: vendors don't publish limits in tokens.

Anthropic describes Claude limits in terms of sessions and weeks ([help center](https://support.claude.com/en/articles/11647753-how-do-usage-and-length-limits-work)). A session lasts five hours: the clock starts with the first message, and after five hours the limit resets fully. Weekly limits across all models apply on top, and some plans have a separate weekly limit for Fable. Usage depends on the length and complexity of the conversation, the model, the effort level, and the features used. Plans (Pro, Max, Ultra, Team) differ in allowance, but none of it is expressed in tokens.

For ChatGPT Plus, OpenAI's help center publishes the current limits. There are no numbers here: subscription terms change, and token figures from third-party reviews aren't confirmed by the vendor. If you need predictable token consumption, that's an argument for the API, where every response comes with a `usage` field.

## How to save tokens

The full economics of an LLM task (a cost ledger, cost per successful task, routing, semantic caching) is covered in the article on [AI cost optimization](https://futurecraft.pro/blog/ai-cost-optimization/). What moves the token count the most:

1. **Limit output.** Output costs 5x input, so `max_tokens` (Anthropic) or `max_output_tokens` (OpenAI) plus a request for brief answers saves more than trimming the prompt. A structured answer against a JSON schema is usually shorter than prose.
2. **Watch thinking.** On reasoning models, thinking is billed at the output rate. On Claude, the effort parameter sets the depth; per the docs, at low effort the model may skip thinking on an easy request entirely.
3. **Cache the stable prefix.** Put the system prompt and reference documents at the start of the request and the variable part at the end. A cache read on Opus 5.5 and Sonnet 5.5 costs 5% of the input price.
4. **Don't resend what isn't needed.** Remove duplicate context, compress old history into a summary, and send retrieved document chunks instead of whole files. The [context engineering guide](https://futurecraft.pro/blog/context-engineering-guide/) covers context management in detail.
5. **Pick the model per task.** Send classification and field extraction to Haiku 5.5 or GPT-6 Luna, complex reasoning to Opus 5.5 or GPT-6 Astra. The input price differs 40-100x.
6. **Send non-urgent work to the Batch API.** It's 50% off at Anthropic, OpenAI, and Google.
7. **Verify with usage.** Check every optimization against the `usage` field before and after, not against the word count of the prompt.

## Common token counting mistakes

- **Treating a token as a word.** In the measurement, an English word took 1.12-1.7 tokens and a Russian word 1.8-2.9, depending on the model. The estimate "1,000 words = 1,000 tokens" undercounts.
- **Assuming "4 characters per token" everywhere.** In the measurement, English prose came to 4.5-5.2 characters per token on most tokenizers and 3.2-3.6 on Claude Haiku 5.5. Russian prose ranged from 2.4 (Claude Haiku 5.5) to 4.1 (Gemini 3.8 Flash).
- **Forgetting history and tools.** In a chat, every turn resends the full history. In the author's measurement on Haiku 5.5, the input of the tenth turn was 57x the first: 1,134 tokens against 20. Tool schemas are also part of the input on every request.
- **Carrying counts over between models.** A tiktoken count doesn't apply to Claude, and a count from older Claude models doesn't apply to 4.7 and later.
- **Using `encoding_for_model` for GPT-6.** tiktoken has no mapping for `gpt-6`, so the call fails with `KeyError`. Set the encoding explicitly and treat the result as an estimate.
- **Setting the output limit too low on a reasoning model.** At OpenAI, reasoning counts toward `max_output_tokens`. If the limit runs out during reasoning, the response comes back `incomplete` with no visible text, and you still pay for input and reasoning.

## Where to start

1. Take 2-3 typical texts from your product and count them with the tokenizer of the model you use: `count_tokens` for Claude, tiktoken `o200k_base` as an estimate for GPT.
2. If your product serves more than one language, count the same texts in each language and get your own ratio instead of someone else's coefficient.
3. Log `usage` for every response, with input, output, thinking, and cache tracked separately.
4. Multiply tokens by the provider's prices with a snapshot date, and compare models by cost per task, not by price per million.
5. Start saving with the output limit and prefix caching, and check the effect in `usage`.

## Primary sources

- [Anthropic: Token counting](https://platform.claude.com/docs/en/build-with-claude/token-counting)
- [Anthropic: Pricing](https://platform.claude.com/docs/en/about-claude/pricing)
- [Anthropic: Models overview](https://platform.claude.com/docs/en/models/overview)
- [Anthropic: Vision](https://platform.claude.com/docs/en/build-with-claude/vision)
- [Anthropic: Extended thinking](https://platform.claude.com/docs/en/build-with-claude/extended-thinking)
- [Anthropic: How do usage and length limits work](https://support.claude.com/en/articles/11647753-how-do-usage-and-length-limits-work)
- [OpenAI: Pricing](https://developers.openai.com/api/docs/pricing)
- [OpenAI: Reasoning models](https://developers.openai.com/api/docs/guides/reasoning)
- [openai/tiktoken](https://github.com/openai/tiktoken)
- [Google: Tokens](https://aistudio.google.com/docs/tokens) and [Pricing](https://aistudio.google.com/docs/pricing)
- [DeepSeek: Pricing](https://api-docs.deepseek.com/quick_start/pricing/)
- [OpenRouter: Usage accounting](https://openrouter.ai/docs/use-cases/usage-accounting)
- [Ollama: API](https://github.com/ollama/ollama/blob/main/docs/api.md)
- [Sennrich, Haddow, Birch. Neural Machine Translation of Rare Words with Subword Units (2015)](https://arxiv.org/abs/1508.07909)
- [Dobrovolskyi. Beyond Two Bytes per Letter: Tokenization Overhead in Cyrillic AI Systems (2026)](https://arxiv.org/abs/2608.21384)