LLM rate limit backoff playbook

Rate limits are not outages, but they feel like outages if you retry blindly. A good backoff playbook protects your budget, your latency SLOs, and your neighbors on shared quotas. This is ops guidance for product features that call LLM APIs — not a vendor-specific runbook.

Related: LLM cost control for teams, JSON schema output prompts, token estimator.

What you are optimizing

Tradeoffs to state explicitly:

• Success rate under burst traffic
• Cost (retries burn tokens on some failure modes; always burn engineer time)
• Latency (users hate 30s silent waits)
• Fairness (one noisy tenant should not starve others)

If you only maximize success rate, you will build an accidental DDoS against your own quota.

Detect the right errors

Distinguish:

• 429 / rate limit — slow down; retry with backoff if budget remains
• 5xx / timeout — retry cautiously; may be model overload
• 4xx validation / policy — do not retry the same payload
• Schema parse failures — retry only with a bounded validator-feedback loop

Map vendor error codes to your internal taxonomy. Log error_class, model, tenant, and retry_count.

Exponential backoff with jitter (default recipe)

A widely used pattern:

1. Attempt 1 immediate
2. On retryable failure: wait base * 2^attempt with full jitter (random between 0 and that cap)
3. Cap wait (e.g. 30–60s) and cap attempts (e.g. 3–5)
4. Honor Retry-After headers when present — they beat your formula

Jitter prevents synchronized stampedes when many workers wake at once. Exact constants belong in config, not folklore.

Separate retry budgets from product budgets

Give each request a retry budget:

• Max attempts
• Max additional wall-clock
• Max extra tokens / $ you are willing to spend on recovery

When the budget is exhausted: fail fast, show a degradated UX, enqueue for later, or route to a fallback model — depending on the feature. Spreadsheet the expected retry multiplier into monthly forecasts (token cost estimation).

Queues beat hot loops

For non-interactive work (batch summarization, night-time embedding):

• Push jobs to a queue with concurrency tied to measured RPM/TPM headroom
• Use token bucket or leaky bucket client-side
• Prefer longer queues over aggressive parallel chat sessions

For interactive UX:

• Show progress / “still working” states
• Offer “notify me” for slow paths
• Cache identical prompts where safe

Multi-model and multi-key tactics (use carefully)

Fallbacks can help availability but complicate privacy and quality:

• Document which models are allowed fallbacks
• Do not fall back to a model that trains on prompts if the primary was zero-retention
• Split API keys per environment and per high-volume feature for blast-radius control

Never multiply keys solely to evade contractual rate limits — that is a policy problem, not an engineering flex.

Client libraries and idempotency

• Make writes idempotent where the product allows (dedupe keys on “generate blog draft”)
• Avoid retrying non-idempotent side effects (charging, sending email) without a ledger
• Centralize LLM HTTP in one client so backoff policy is not copy-pasted wrong

User-facing degradation map

Write what the UI does for each class:

Condition	UX
Soft rate limit	Spinner + retry once quietly
Hard quota	Clear message + ETA / upgrade path if applicable
Validator fail	Ask user to shorten input; do not infinite-loop
Provider outage	Fallback copy + status link

Silent spinners with unbounded retries train users to refresh — which worsens limits.

Load-shedding and feature flags

When TPM is near ceiling:

• Disable expensive agent tools first
• Reduce RAG k temporarily (RAG cost estimation)
• Shed batch traffic in favor of interactive
• Flip a flag rather than hot-patching backoff constants at 2 a.m.

Playbook checklist

• Error taxonomy mapped from vendor codes
• Exponential backoff + jitter + Retry-After support
• Per-request retry budget (attempts, time, $)
• Queue concurrency aligned to RPM/TPM
• Metrics: 429 rate, retry success, p95 wait, $ on retries
• Degradation UX documented
• Fallback model policy reviewed for privacy
• Load-shed flags tested in staging

What not to do

• Retry parse errors with the identical prompt forever
• Sleep a fixed 5s on every failure (thundering herd)
• Hide rate limits behind infinite client spinners
• Ignore that streaming aborts and tool loops also consume quota

Rate limits are a signal to shape traffic. Back off with jitter, budget the retries, and degrade on purpose — that is how LLM features stay boring under load.

Tie cost forecasts to retry policy

Before raising concurrency, re-run the token estimator with an explicit retry percentage that matches your client budget. A feature that looks cheap at 0% retries can double in cost under a noisy schema validator. Document the assumed retry rate beside the RPM target so finance and on-call share one number.