Reduce prompt tokens without losing quality

Token reduction is not “make the prompt shorter until it breaks.” It is removing text that does not change model behavior, then verifying on real tasks. Done well, you cut spend, leave room for retrieval, and make prompts easier to review. Done poorly, you delete the constraints that kept the model honest.

Pair this with the prompt cleaner, token estimator, and count tokens without an API key.

What “quality” means here

Define quality before you compress:

• Task success rate on a fixed eval set (even 20 cases)
• Refusal / safety behavior you care about
• Output schema validity (JSON parse rate, required fields)
• Latency and cost per successful job

If you only measure character count, you will ship shorter failures.

Cut noise first (usually free wins)

Before rewriting instructions, scrub junk:

• Chat UI crumbs (“Copy”, “Regenerate”, timestamps, “You said:”)
• Duplicate paragraphs pasted from older versions
• Soft adjectives that do not constrain behavior (“be insightful”, “world-class”)
• Repeated brand slogans that belong in a short style line once

Run messy dumps through the prompt cleaner so humans review structure instead of whitespace.

Separate durable context from the task

Models (and humans) waste tokens when facts, policy, and the current ask are blended. Prefer labeled blocks:

CONTEXT: (facts only, no verbs of desire)
POLICY: (hard rules, refusals, PII)
TASK: (this turn's job)
OUTPUT: (schema / length / forbid list)

Then shrink CONTEXT with retrieval or links instead of pasting the whole wiki every call. See RAG context window cost estimation when context is retrieved, not static.

Compress examples without deleting the lesson

Few-shot examples are expensive. Keep them only when style or edge cases need demonstration.

Techniques:

• One short positive example beats three long ones
• Prefer “input → output” pairs over narrative walkthroughs
• Drop examples that only restate the schema
• For structured tasks, a tiny schema + one edge case often beats five happy paths

If examples exist only to teach tone, try a 2–3 line tone contract and re-measure.

Prefer schemas over prose constraints

Long paragraphs listing field rules tokenize poorly and invite contradiction. Replace with:

• A minimal JSON Schema or TypeScript-shaped example
• Explicit “UNKNOWN if missing” rules in one line
• A short forbid list (no fake URLs, no invented metrics)

More patterns: JSON schema output prompts for production.

Route length to the right model tier

Sometimes the “compression” is architectural:

• Classify / extract with a small model and short prompt
• Escalate only hard cases to a larger model with richer context
• Cache stable system prefixes when your vendor supports prompt caching

Token reduction and LLM cost control for teams belong in the same RFC.

A safe reduction loop

1. Baseline — measure tokens (offline or usage) and quality on the eval set.
2. Hypothesize — name what you will delete and why it should be safe.
3. Change one axis — noise scrub, example cut, or schema swap — not all at once.
4. Re-measure — tokens, parse rate, and task success.
5. Keep a diff — prompt versions like config; see production prompt hygiene in production system prompts.

Never compress and ship in the same PR without an eval note.

What not to delete

Keep (or shorten carefully) text that encodes:

• Hard safety / compliance refusals
• “Do not invent citations / numbers” rules
• Tool-use contracts and stop conditions
• Output length caps that prevent runaway completions
• Domain facts that are not available via retrieval

If a line prevents a known failure mode, it is not fluff.

Quick wins checklist

• Cleaner run on pasted sources
• CONTEXT / POLICY / TASK / OUTPUT separated
• Examples reduced to the minimum that passes eval
• Prose field lists replaced with a schema sketch
• p90 RAG size accounted for separately from the system prompt
• Token delta and quality delta recorded next to the prompt version

Shrinking prompts is an ops habit, not a one-time rewrite. Revisit when models, tools, or product claims change — and re-estimate spend with the token estimator before you celebrate the savings.

Example: before / after (illustrative structure)

Before (noisy): a pasted Slack thread, three overlapping “be concise but thorough” lines, a full style guide, and two long happy-path examples.

After (structured): ROLE one-liner → POLICY five bullets → TASK with success criteria → OUTPUT schema sketch → one edge-case example.

You should expect large character drops from noise removal alone; quality changes only show up on the eval set. Record both.

Team habits that keep prompts lean

• Owner per prompt — someone merges changes and runs the eval note.
• Size budget in the PR template — “estimated tokens before/after”.
• Quarterly garbage collection — delete dead examples and obsolete product names.
• Shared scrubber — agree that messy imports go through the prompt cleaner before review.

Compression without ownership regresses within a month.