Local tokenizer vs vendor tokenizer: differences that matter

See also / canonical cluster. Pillar for token & cost estimation: How to estimate LLM token costs before you ship. Interactive: token & cost estimator. This page stays on a different intent — see the intro.

Intent here: why local vs vendor tokenizer counts diverge and how to document the error band — not monthly cost estimation. Offline counting methods: count tokens without an API key. Concepts: token counting explained.

Two different jobs

Method	Job	Source of truth for
Local / open tokenizer	Plan, CI gates, offline privacy	Engineering estimates
Vendor usage fields	Billing and quotas	Invoices and rate limits
Character heuristics	Back-of-envelope	Early ideation only

If you conflate the jobs, you will either over-trust a library or refuse useful offline tooling.

Why counts diverge

Common causes:

1. Different vocabularies — model families tokenize the same UTF-8 bytes differently.
2. Special tokens — chat templates add role markers, stop tokens, tool headers you did not paste into the local string.
3. Pre/post processors — normalization, BPE dropout (training), or undocumented merging rules.
4. Tool / vision / audio wrappers — “text tokens” are only part of the billable story.
5. Stale local files — you pinned tiktoken encoding A while production moved to encoding B.
6. Counting the wrong string — local count on user text only; vendor counts system + tools + memory.

A 5–15% band on English prose is not unusual when templates differ. Code, JSON, and CJK can skew more.

When the gap is harmless

Offline counts are “good enough” when:

• You need order-of-magnitude monthly cost
• You enforce a soft CI ceiling with margin (e.g. fail at 80% of hard context)
• You will calibrate with a tiny live sample before launch

They are not good enough when:

• You are within a few hundred tokens of a hard context limit
• You bill customers per token and need invoice-grade meters
• You compare two vendors’ “1K tokens” as if units were identical — see also mini vs Haiku cost worksheet

A calibration workflow

1. Freeze the exact production payload shape (system, tools, messages).
2. Count locally with the tokenizer that claims to match the family.
3. Send one short live call (or a dry-run usage endpoint if offered).
4. Record local_count, vendor_usage, ratio = vendor/local.
5. Store ratio, tokenizer package version, and model id next to the prompt version.

Re-run calibration when you change model ids, chat templates, or tool schemas.

Documenting the error band

In the feature RFC, write:

Tokenizer: <package>@<version> / encoding <name>
Calibrated: <date> against model <id>
Band: vendor usage ≈ local * <ratio> (± <pct>% on n=<samples>)
Budget uses: vendor-adjusted estimate
CI uses: local count with <margin> headroom

This prevents a six-month-later argument about why the dashboard disagrees with CI.

Heuristics vs local libraries vs vendor

• Heuristics (~4 chars/token) — fine in the token estimator for workshops; label as planning.
• Local libraries — best offline gate when encoding matches the family.
• Vendor usage — only number finance should settle against.

For RAG packs, count retrieved chunks + templates, not just the user question. Context packing errors dwarf tokenizer skew — see RAG context window cost estimation.

CI patterns that survive disagreement

• Fail PRs when local count > threshold × 0.8 (leave margin)
• Optionally fail when estimated vendor-adjusted count > hard limit
• Do not call paid APIs from CI just to tokenize unless you accept the secret and cost surface
• Cache calibration ratios in repo config, reviewed like dependency bumps

Anti-patterns

• Asserting “tokens are tokens” across vendors in a cost bake-off
• Updating models but not tokenizer pins
• Counting Markdown files on disk while production sends JSON tool traces
• Building customer invoices from local counts without a reconciliation job

Practical next steps

1. Pick one production prompt and calibrate local vs vendor today.
2. Add tokenizer version to your prompt changelog — see document prompt changes like config.
3. Use heuristics only with explicit labels in planning sheets.
4. Keep deep links to count tokens without an API key for teammates who need the overview.

Local tokenizers are instruments. Vendor usage is the register. Good teams keep both and write down the delta.

Worked example

You count 1,000 local tokens on a user+system string. Production usage returns 1,120 prompt tokens because the chat template added role markers and a tool preamble. Ratio 1.12. Your CI gate that fails at 8,000 local tokens is roughly a 9,000 vendor-token ceiling. Document vendor ≈ 1.12 * local in the prompt manifest and revisit when templates change.

Multi-vendor products

If you route the same prompt to two vendors, keep two calibration rows. Never convert vendor A usage into vendor B cost with a single tokenizer assumption — billable tokens are not a shared currency.