Token counting explained for builders and budget owners

Interactive tool: Need an immediate calculation? Use our free, browser-based token & cost estimator — calculate prompt tokens and monthly API spend client-side with editable $/1M rates and zero API key requirement.

Canonical cluster: Pillar for pre-ship spend forecasts: How to estimate LLM token costs before you ship.

This is the comprehensive fundamentals and methods guide: what tokens are, how to count tokens offline without API keys, why local and vendor tokenizers diverge, how to calibrate error bands for CI, and how counting relates to context windows and production budgets.

---

What is a token?

A token is a chunk of text produced by a tokenizer — often a subword piece. Rough intuition for English prose:

• ~4 characters per token
• ~0.75 words per token (or ~100 tokens ≈ 75 words)

These are rules of thumb, not laws. Code, URLs, non-English text, and whitespace-heavy JSON tokenize differently.

Chinese often uses fewer characters per token than a naive English rule suggests — still measure with a real tokenizer when precision matters.

---

Two different jobs: Local estimation vs vendor billing

If you conflate these jobs, you will either over-trust a library or refuse useful offline tooling. Offline estimators (including ours) approximate for planning. Vendor usage.prompt_tokens in API responses remains the source of truth for billing.

---

Why count without calling the API?

• Budgeting — Finance wants a range before you enable a feature.
• Design — You are choosing between short system prompts and long RAG packs.
• Privacy — Sample prompts contain customer text you should not send to a vendor yet.
• CI — You want a gate that fails oversized prompts without network access or secret tokens in runners.

---

Offline counting methods (no key required)

Method 1: Character / word heuristics

For English prose, a common rule of thumb is ~4 characters per token, or ~0.75 words per token. That is enough for rough monthly forecasts when you multiply by call volume and retries.

Use heuristics when:

• You need an order-of-magnitude answer in five minutes.
• The text mix is mostly natural language (not dense code or JSON).
• You will validate later with a real tokenizer or a small live sample.

Do not treat heuristics as invoice-grade. Code, URLs, Markdown tables, and CJK text skew the ratio. Our token & cost estimator exposes an editable chars-per-token field so you can stress-test assumptions directly in your browser.

Method 2: Local / open tokenizers

Many model families publish or mirror tokenizer files you can run offline (library packages, WASM builds, or CLI tools). Local counting is the right move when:

• You already know which model family you will call.
• Prompt size sits near a hard context limit.
• You want CI to reject oversized templates without secrets in the runner.

Caveats:

• Tokenizer must match the family you bill against (or accept a known error band).
• Updating models means updating the local vocab — pin versions in docs.
• Some vendors do not ship a public tokenizer; then approximate and sample-validate.

Document which tokenizer artifact you used next to the prompt version. Treat it like a dependency.

Method 3: Browser estimators (no key, no upload)

Client-side estimators (including the sudoai.net token estimator) keep text in the browser, apply a simple rule, and pair it with editable USD rates. Good for:

• Writers and PMs who will not install Python packages.
• Quick “what if we cut this section” comparisons.
• Workshops where you cannot share API keys.

They are not substitutes for vendor usage fields. Label estimates as planning numbers in RFCs.

Method 4: Tiny paid sample, then extrapolate

If you have a key but want to minimize spend:

1. Tokenize or estimate offline to get a baseline.
2. Send one short representative call.
3. Compare usage.prompt_tokens to your estimate.
4. Scale the correction factor across the full corpus.

That single calibration call is often cheaper than debating ratios for an hour.

---

What consumes tokens in a request?

Typical chat/completions request:

1. System message
2. Developer / tool instructions (if any)
3. Conversation history
4. Retrieved documents / RAG chunks
5. Tool schemas and tool results
6. The new user message
7. Output tokens from the model reply (billed separately, often at a different rate)

Agents multiply (4)–(6) across loop iterations. That is why research agents need max-call budgets (see system prompts for research agents).

---

Context window vs cost

• Context window — hard maximum tokens the model can consider (input + output constraints vary by API).
• Cost — usually dominated by how many tokens you actually send and generate, not the maximum window.

A 128k window does not mean you should send 128k tokens. More context is not free and often hurts quality when noisy.

---

Why your estimator and local counts diverge from vendor APIs

Divergence between offline counts and vendor invoices boils down to six root causes:

1. Different vocabularies — model families tokenize the same UTF-8 bytes differently.
2. Special tokens — chat templates add role markers, stop tokens, and tool headers you did not paste into the local string.
3. Pre/post processors — normalization, BPE dropout (training), or undocumented merging rules.
4. Tool / vision / audio wrappers — “text tokens” are only part of the billable story.
5. Stale local files — you pinned tiktoken encoding A while production moved to encoding B.
6. Counting the wrong string — local count on user text only; vendor counts system + tools + memory.

A 5–15% band on English prose is not unusual when templates differ. Code, JSON, and CJK can skew more.

---

When the gap is harmless vs when it matters

Offline counts are “good enough” when:

• You need order-of-magnitude monthly cost.
• You enforce a soft CI ceiling with margin (e.g. fail at 80% of hard context).
• You will calibrate with a tiny live sample before launch.

They are not good enough when:

• You are within a few hundred tokens of a hard context limit.
• You bill customers per token and need invoice-grade meters.
• You compare two vendors’ “1K tokens” as if units were identical — see also our Luna vs Haiku 4.5 cost worksheet.

---

A calibration workflow & worked example

1. Freeze the exact production payload shape (system, tools, messages).
2. Count locally with the tokenizer that claims to match the family.
3. Send one short live call (or a dry-run usage endpoint if offered).
4. Record local_count, vendor_usage, ratio = vendor/local.
5. Store ratio, tokenizer package version, and model id next to the prompt version.

Re-run calibration when you change model ids, chat templates, or tool schemas.

Worked example

You count 1,000 local tokens on a user+system string. Production usage returns 1,120 prompt tokens because the chat template added role markers and a tool preamble. Ratio is 1.12. Your CI gate that fails at 8,000 local tokens is roughly a 9,000 vendor-token ceiling. Document vendor ≈ 1.12 * local in the prompt manifest and revisit when templates change.

Documenting the error band

In the feature RFC, write:

Tokenizer: <package>@<version> / encoding <name>
Calibrated: <date> against model <id>
Band: vendor usage ≈ local * <ratio> (± <pct>% on n=<samples>)
Budget uses: vendor-adjusted estimate
CI uses: local count with <margin> headroom

This prevents a six-month-later argument about why the dashboard disagrees with CI.

---

Multi-vendor products

If you route the same prompt to two vendors, keep two calibration rows. Never convert vendor A usage into vendor B cost with a single tokenizer assumption — billable tokens are not a shared currency.

---

CI patterns that survive disagreement

• Fail PRs when local count > threshold × 0.8 (leave margin).
• Optionally fail when estimated vendor-adjusted count > hard limit.
• Do not call paid APIs from CI just to tokenize unless you accept the secret and cost surface.
• Cache calibration ratios in repo config, reviewed like dependency bumps.

---

Counting for non-English and code

• CJK text: do not assume 4 chars/token; sample real counts.
• Code: operators and identifiers can be token-heavy; indentation matters less than you think, but long minified lines can be worse.
• Base64 / hex: extremely token-inefficient — avoid stuffing blobs into prompts.
• Markdown tables: usually fine; huge HTML dumps are not.

---

Output token control

Output is often pricier per token than input on frontier models.

• Set max_tokens to the task (titles vs essays).
• Ask for compact formats (bullets, JSON without comments).
• Forbid restating the user question.
• For creative tasks, accept higher caps deliberately — and budget them.

---

Common mistakes and anti-patterns

• Counting only the user message and forgetting the system prompt, tool schemas, and conversation history.
• Asserting “tokens are tokens” across vendors in a cost bake-off.
• Using GPT-era rules for a different vendor’s tokenizer.
• Updating models but not tokenizer pins.
• Ignoring whitespace-heavy JSON and base64 blobs in context.
• Estimating once on a happy-path sample and never checking the p90 case.
• Counting Markdown files on disk while production sends JSON tool traces.
• Building customer invoices from local counts without a reconciliation job.
• Shipping without a documented chars-per-token or tokenizer version.

---

Practical estimation workflow & pre-launch checklist

Practical estimation workflow

1. Freeze the prompt shape — system + user template + typical RAG chunk sizes.
2. Estimate offline — heuristic or local tokenizer on 3–5 real examples (p50 / p90).
3. Add output headroom — max_tokens and typical completion length, not just input.
4. Fold in retries and agents — multi-step tools multiply input tokens; see rate limit backoff.
5. Convert to money — test in the token & cost estimator with current invoice rates.
6. Gate in CI — fail PRs that push system prompts past an agreed ceiling.

Do this per feature, not as one blob.

Pre-launch checklist

• Primary feature has a written prompt template.
• Offline estimate method named (heuristic vs local tokenizer).
• p50 and p90 samples measured.
• Output and retry multipliers included.
• Rates match current vendor list prices (edited in the estimator).
• CI or review gate exists for prompt size regressions.

---

Token budgets in product design

Examples:

• Free tier: 50k input tokens / day / user
• Support bot: hard cap 8k context; summarize older turns
• Doc assistant: top-5 chunks ≤ 3k tokens combined
• Eval suite: separate budget so it cannot starve production

See LLM cost control for teams.

---

Teaching teammates a shared language

Print this on the wiki:

• “We’re not short on ideas; we’re short on tokens and edit time.”
• “Expanding the system prompt by 1k tokens at 1M calls/month is a budget change — review it.”
• “Estimator ≠ invoice; API usage = invoice.”

---

Quick lab (10 minutes)

1. Take a 500-word blog paragraph; estimate tokens at 4 chars/token.
2. Run the same text through your provider’s tokenizer or a library when available.
3. Compare error %.
4. Repeat with 50 lines of Python and a Chinese paragraph.
5. Update your team’s rule of thumb per content type.

---

Related tools and guides

• Token & cost estimator — interactive offline counter with editable USD rates.
• Prompt cleaner — remove chat residue and reduce junk tokens before counting.
• LLM token cost estimation (cluster pillar) — pre-ship monthly cost forecasting.
• Reduce prompt tokens without losing quality — prompt compression techniques.
• RAG context window cost estimation — budgeting retrieval packs.
• Document prompt changes like config — versioning prompts in git.
• Prompt engineering checklist — production prompt contracts.