Retry budget for LLM features: a practical spreadsheet

Retries feel free until the invoice lands. An LLM retry budget is an explicit cap on how much extra latency, token spend, and queue depth you allow when calls fail or return unusable output. This guide gives you a spreadsheet-shaped model you can paste into an RFC.

Related: LLM rate limit backoff playbook, token estimator, LLM cost control for teams.

What a retry budget is

A retry budget answers:

• How many automatic retries per user action?
• Which errors are retryable (429, 503, timeouts) vs not (400 validation, policy blocks)?
• What is the max extra spend per day from retries?
• When do we shed load instead of retrying?

Without those answers, every client invents for i in range(5) and your p99 latency becomes a random walk.

Spreadsheet columns that matter

Create one row per LLM feature (not per endpoint if one endpoint serves many products):

Field	Example
Feature	support-draft-v3
Calls / day (p50 / p90)	80k / 140k
Base tokens in+out	1.8k
Retryable error rate (baseline)	1.5%
Max retries / call	2
Backoff policy	exp + jitter, cap 8s
Expected retry multiplier	1.03
Daily token $ (base)	…
Daily token $ (with budget)	…
Latency SLO p95	4s
Fail mode	degrade to template

Retry multiplier (simple):

1 + retryable_rate * avg_retries_used

If 1.5% of calls use 1.2 retries on average: 1 + 0.015 * 1.2 ≈ 1.018. Under incidents the rate spikes — budget for a bad day column (e.g. 10% retryable) so finance is not surprised.

Retryable vs non-retryable

Usually retryable (with backoff):

• HTTP 429 / rate limit
• 502 / 503 / 504 from gateway
• Transient network timeouts
• Empty body with 5xx

Usually not retryable:

• 400 bad request / schema client bugs
• Auth failures
• Content-policy hard blocks (unless you have a safe rewrite path)
• “Model returned JSON that failed validation” — may allow one repair pass, counted separately

Treat repair passes (send error + ask to fix JSON) as a different budget line from transport retries. They burn tokens even when the HTTP layer succeeded.

Linking retries to money

1. Estimate base monthly cost in the token estimator.
2. Add retry_rate (the tool already has a retry % field).
3. Add a second scenario: “incident day” at 5–10× retryable rate.
4. Cap auto-retries so incident day cannot exceed a dollar ceiling without a human flipping a kill switch.

Example narrative for stakeholders: “Base plan assumes 2% retryable × 1 retry ≈ +2% spend. Kill switch trips if retry spend exceeds 8% of daily budget.”

Latency and UX constraints

A budget that is only about dollars will still fail users:

• Cap total wait (e.g. retries must finish inside the p95 SLO).
• Prefer fail fast + queue for batch jobs; prefer immediate degrade for interactive UI.
• Show honest UI (“Still working…” with cancel) instead of silent multi-retry loops.

Coordinate with the rate limit backoff playbook so jitter and shared quotas do not amplify thundering herds.

Spreadsheet formulas (copy-friendly)

Assume:

• B = base cost per successful call
• r = fraction of calls that hit at least one retry
• n = average retries among those calls
• f = fraction of retries that are full re-runs (pay full tokens again)

Then:

expected_cost_per_user_action ≈ B * (1 + r * n * f)

If repair prompts are shorter, use a separate B_repair term instead of B.

Track success after retries separately from HTTP success — a 200 with unusable JSON still costs money.

Operational checklist

• Retryable error taxonomy documented in the service README
• Max retries + max wall-clock encoded in client middleware
• Metrics: retry count, retry spend $, success after retry
• Alert when retry spend % exceeds threshold
• Kill switch / feature flag to disable non-critical retries
• Incident runbook links to backoff settings

Anti-patterns

• Retrying on every exception including programmer bugs
• Unlimited retries in serverless with long timeouts (surprise bill)
• Same retry policy for interactive chat and nightly batch
• Counting only provider 429s while ignoring application-level repair loops
• No link between retry metrics and the cost dashboard

Next steps

1. Add a retry-budget row for each LLM feature this week.
2. Wire retry % into the token estimator forecast you show in planning.
3. Align middleware with the backoff playbook.
4. Fold expected retry multiplier into unit economics.

A retry budget is a product decision written as numbers — not an afterthought buried in a shared HTTP client.