RAG context window cost estimation

See also / canonical cluster. Pillar for token & cost estimation: How to estimate LLM token costs before you ship. Interactive: token & cost estimator. This page stays on a different intent — see the intro.

Intent here: RAG / context-window cost shape (chunk packs, retrieval multipliers) — not the general feature estimation checklist. Pair with the estimation pillar and the token estimator.

The RAG cost stack

Per call, tokens roughly equal:

1. System / developer prompt (policies, tool instructions)
2. Conversation history (if you keep turns)
3. Retrieved chunks (top-k × average chunk tokens)
4. User question (usually tiny)
5. Output (answer + citations + tool chatter)

For many production RAG apps, (3) dominates. Agents that re-retrieve each step multiply (3) by turns.

Measure chunks, not vibes

Pick 20–50 real queries (support tickets, docs questions, internal search logs). For each:

• Record how many chunks your retriever returns at the planned k
• Measure token counts with a heuristic or local tokenizer (count without an API key)
• Note empty / low-relevance retrieves — they still cost if you stuff them

Report p50 and p90 input tokens, not only the mean. Billing follows the fat tail when traffic spikes on hard queries.

A workable formula

input_tokens ≈ system + history + (k * avg_chunk_tokens) + question
output_tokens ≈ expected_answer + citation_overhead
cost_per_call = input_tokens/1e6 * in_rate + output_tokens/1e6 * out_rate
monthly ≈ calls * cost_per_call * (1 + retry_rate) * agent_turns

Plug the p90 input_tokens into the token estimator when you want a conservative forecast. Edit model rates to match your vendor invoice — defaults on the tool are placeholders.

Levers that actually move cost

Lower k — Fewer chunks cut tokens linearly if quality holds. Validate with your eval set before celebrating.

Smaller chunks / smarter packing — Overlapping huge chunks duplicate tokens. Prefer retrieval that returns dense, non-redundant spans.

Rerank then pack — Retrieve wider, keep only the top spans after a cheap rerank. You pay a small rerank cost to avoid stuffing garbage into the expensive generative call.

Prompt compression — Shrink the static system prompt (reduce prompt tokens) so the window belongs to retrieved evidence.

Model routing — Easy FAQ hits on a small model; escalate ambiguous cases with larger context to a mid/large model.

Caching — Cache embeddings and, where supported, cache stable system prefixes. Do not invent cache hit rates — measure after launch.

Context window ≠ free budget

A 128k window does not mean you should fill it. Cost scales with tokens billed, and quality often drops when you paste weakly related chunks. Treat the window as a ceiling, and design a target pack size (for example, a p90 pack well under the limit) as a product decision.

Document:

• Max chunks and max pack tokens
• What happens when retrieval exceeds the pack (truncate? refuse? summarize?)
• Citation requirements that force extra output tokens

Retries, tools, and multi-hop

Inflators to include in the spreadsheet:

• Retry / fallback rate when JSON or tools fail
• Tool-calling overhead (schemas + results echoed back)
• Multi-hop retrieval (research agents)
• Streaming abandoned mid-answer (you may still pay for partial output depending on vendor)

For backoff and quota behavior under load, see LLM rate limit backoff playbook.

Minimal spreadsheet columns

Column	Why
Query id	Traceability
k	Retrieval width
Chunk tokens (sum)	Dominant input
System + history tokens	Often forgotten
Output tokens (est.)	Completions bill separately
Model	Rate table key
Retries / turns	Multipliers
$/call and $/month	Decision output

Keep one tab for current production and one for proposed packing so reviews compare apples to apples.

Ship checklist

• p50 / p90 pack sizes measured on real queries
• System, history, and tool schema tokens included
• Rates match current vendor pricing
• Max pack policy written into the system prompt / service config
• Eval proves quality at the cheaper pack size
• Alerts on average input tokens per successful answer

RAG cost control is mostly retrieval discipline plus honest forecasting. Estimate the pack, not the question — then wire alerts so context creep cannot silently rewrite your unit economics.

Worked sketch (numbers are placeholders — replace with yours)

Suppose p90 pack is 6 chunks × 400 tokens = 2,400, system+tools = 800, question = 40, history = 200 → ~3,440 input tokens. Expected output 500 tokens. At illustrative mid-tier rates of $0.50 / $1.50 per 1M input/output:

• Input cost/call ≈ 3440/1e6 × 0.50
• Output cost/call ≈ 500/1e6 × 1.50
• With 8% retries and 120k calls/month, multiply accordingly in the token estimator

Do not publish these placeholder rates as market truth — edit the tool to your invoice and re-run when vendors change price sheets.

Quality vs pack size experiments

Run a simple ladder on the same query set: k=3, k=5, k=8 (or fixed token budgets). Plot:

• Answer usefulness / citation precision (human or rubric)
• Input tokens and $/1k successful answers
• Latency

Pick the knee where quality flattens but cost still rises. That knee is your default pack policy — write it into config, not tribal knowledge.