Token counting is wrong: measure actual LLM cost

Your bill doesn't match your token estimates, and the gap is widening. Counting prompt and completion characters gives you a number that ignores retries, cached tokens, tool definitions, and the asymmetric cost of output. The fix is to instrument actual usage from the API response and measure what you're actually charged for.

Why your token estimate diverges from the bill

The root cause is that tokenizers are model-specific. The same string of text can produce different token counts across models — a block of JSON that tokenizes to 150 tokens in one model might be 180 in another. If you're estimating with a generic character-to-token ratio (say, 4 characters per token), you'll drift badly the moment your payload includes code, JSON, or non-English text. Code and structured data tend to tokenize less efficiently than natural language because the tokenizer wasn't trained on much of it.

Hidden costs compound the error. Every request carries system prompts, few-shot examples, and tool definitions that are re-tokenized on every call. Most developers estimate only the user prompt and expected completion, ignoring the scaffolding around it. I've seen teams slash their prompt text by 30% only to discover their system prompt was three times larger than the user input all along. That overhead is invisible if you're counting only what you typed into a playground.

Then there are retries. A single failed attempt still consumes tokens — often the full prompt plus a truncated or error completion. If you retry three times before getting a success, your effective cost per successful response is four times the per-call estimate, not counting any speculative decoding overhead. Multi-turn agent loops make this worse: each step carries accumulated context, so a five-turn conversation doesn't cost five times a single turn — it costs fifteen or more as context grows linearly and retries stack. The gap between your mental model and your invoice is systematic, not random.

The input/output split matters more than total tokens

Pricing is almost always asymmetric. Output tokens typically cost two to four times input tokens across major providers. A request with 2,000 input tokens and 1,000 output tokens can cost more than one with 10,000 input tokens and 50 output tokens, even though the total is lower. If you're tracking only total tokens, you cannot tell which scenario you're in.

Measure input and output separately across a production sample, not a single test prompt. A single example tells you nothing about the distribution. I log every API response's usage.prompt_tokens and usage.completion_tokens and aggregate over a thousand requests to get a real ratio. In agent loops, I've seen output token counts double or triple input counts per call because the model generates long tool call arguments and structured responses. That's the expensive regime.

Track tokens per request and per session. Session-level totals reveal context growth that per-call numbers hide. A single turn might show 500 input and 200 output, but turn ten in the same session could be 5,000 input and 800 output as the conversation history accumulates. If you're billing per-session or measuring user-level cost, per-call averages are dangerously misleading. The session total is the number that matters for cost allocation, and it often follows a super-linear curve.

Cache hits turn token math upside down

Prompt caching fundamentally changes the cost equation. Providers charge a fraction of the input token price on cache hits — sometimes 10% or less of the uncached rate. That means the effective cost of a request depends on cache hit rate, not raw token count. Two requests with identical token counts can differ in cost by an order of magnitude if one hits cache and the other doesn't.

Measure cache hit rate per system prompt and per conversation turn. A low hit rate often means your prompt is cache-unfriendly. Common culprits include dynamic timestamps, random nonces, or user-specific metadata injected into the prefix. If every request has a unique prefix, nothing gets cached. I wrote more about diagnosing this in Prompt caching: cache keys, misses, and getting a useful hit rate.

Compare cost with and without caching by simulating the uncached price against your actual logs. The difference can dwarf any token-count optimization you'd make manually. I've seen teams spend a week trimming 10% off their prompt length while ignoring that a 60% cache hit rate would save them 40% on input costs. The math is straightforward: if cached input costs 10% of uncached, a 50% hit rate makes your effective input cost per request 55% of the base (0.5 * 1.0 + 0.5 * 0.1). That's a bigger lever than squeezing characters.

Cache hit rate isn't static. It changes with prompt engineering, model version updates, and traffic patterns. Log it continuously. Anthropic's documentation on prompt caching explains the exact conditions for a cache hit and how to inspect the cache_creation_input_tokens and cache_read_input_tokens fields.

Tool calls and structured outputs inflate token usage

Tool schemas and JSON schema definitions are re-sent on every request. A verbose schema with long descriptions, anyOf unions, or deeply nested objects can dominate input tokens in short exchanges. I've seen a single tool definition with eight parameters and multi-sentence descriptions consume 1,200 tokens — more than the user's actual question. If your conversation is mostly short queries with rich tool schemas, the tools are your primary cost driver.

Each tool call also generates a completion plus a follow-up call where the model decides what to do with the result. That doubles output tokens and adds a round-trip of latency. In agent loops with three or four tool invocations per turn, the token overhead from tool calls can exceed the actual reasoning tokens. Logan token usage per tool invocation and per retry. If a particular tool schema is disproportionately large relative to how often it's used, consider simplifying the descriptions or flattening the structure. I cover this pattern in detail in Building an agent loop: tool calls, retries, failure modes.

Structured outputs add another layer. When you constrain the model with a JSON schema validators on the provider side, the tokenizer has to produce valid JSON tokens in sequence, which can inflate output counts compared to free-form text for the same semantic content. The trade-off is worth it for reliability, but you should measure the overhead. If you're already using JSON Schema, Reliable Structured Outputs from LLMs Using JSON Schema covers how to define schemas that minimize token waste while preserving correctness.

What to instrument instead of token estimates

Record actual token usage from the API response for every request. Every major provider returns a usage object with prompt_tokens, completion_tokens, and total_tokens. Some also include cached token details and tool call breakdowns. Log these fields — not the string length of your prompt — to a structured log sink or a timeseries database. Here's a minimal example for OpenAI:

import openai
import json
 
def log_request_cost(response: openai.types.chat.ChatCompletion) -> dict:
    usage = response.usage
    cost = {
        "model": response.model,
        "input_tokens": usage.prompt_tokens,
        "output_tokens": usage.completion_tokens,
        "cached_input_tokens": (
            usage.prompt_tokens_details.cached_tokens
            if usage.prompt_tokens_details
            else 0
        ),
        "total_tokens": usage.total_tokens,
        "request_id": response.id,
    }
    # Ship to your observability system
    print(json.dumps(cost))
    return cost

Store this per-request, but also compute per-session and per-user aggregates. The API response fields are the single source of truth — your local tokenizer estimate is always a guess.

Add latency per request and time-to-first-token to your logs. Cost and user experience are separate dimensions, and token count alone predicts neither. A request with 10,000 input tokens might stream back output in 200ms or 2 seconds depending on provider load and model size. Track both numeric and timing data.

Track retries and fallbacks explicitly. Every failed attempt consumes tokens. I store a retry_count and fallback_model field on each log entry so I can compute effective cost per successful response. The formula is simple: sum all tokens consumed across retries, then divide by the number of successful responses. That number is frequently 2-5x the per-call average.

Here's a function that calculates effective cost from a log of requests including retries:

def effective_cost_per_success(entries: list[dict], pricing: dict) -> float:
    total_cost = 0.0
    successes = 0
    for entry in entries:
        input_tokens = entry["input_tokens"]
        cached_tokens = entry.get("cached_input_tokens", 0)
        output_tokens = entry["output_tokens"]
        cost = (
            (input_tokens - cached_tokens) * pricing["input"]
            + cached_tokens * pricing["cached_input"]
            + output_tokens * pricing["output"]
        )
        total_cost += cost
        if entry.get("success"):
            successes += 1
    return total_cost / successes if successes else 0.0

Failure modes and gotchas in token measurement

Streaming responses report usage only at the end of the stream. If you log mid-stream — for example, after each chunk — you will miss the final token count and likely undercount output tokens. Always wait for the final stream chunk, which carries the usage field in most provider SDKs. If you're aggregating streaming metrics in real time, buffer the running count from the usage field of the terminal chunk only.

Some providers include cached tokens in the usage object but separate them into subfields like prompt_tokens_details.cached_tokens. If you sum prompt_tokens and cached_tokens naively, you double-count the cached portion. The prompt_tokens field typically includes cached tokens in the total. Treat the cached subfield as a breakdown, not an additive component. OpenAI's API reference documents the exact structure of the usage object.

Token counts can vary slightly between the API's reported values and local tokenizer libraries like tiktoken or transformers. These differences arise from version mismatches or subtle implementation differences in the tokenizer. Treat your local count as an approximation. For billing and cost analysis, always use the API-reported values. If you're doing pre-request estimation for budgeting, calibrate your local tokenizer against a sample of production API responses to measure the drift.

One more thing: if you use a proxy or gateway that aggregates multiple provider calls into a single logical request, make sure you log each underlying API call separately. A single user action might trigger three agent steps, each with its own token usage and cache status. Aggregating at the gateway level without preserving per-call granularity will hide the real cost distribution.

Key takeaways

• Track prompt_tokens and completion_tokens from the API response, not character counts or local tokenizer estimates, for every request.
• Separate input and output costs in your monitoring — asymmetric pricing means total tokens alone is a poor cost proxy.
• Log cache hit rate per system prompt and per session; caching can change your effective input cost by more than any prompt trimming effort.
• Instrument retries, tool invocations, and session-level accumulations explicitly — these hidden multipliers dominate real-world bills.
• Treat local tokenizer counts as approximations and always prefer the API's usage object as the source of truth for cost analysis.