Evaluate LLM vendor lock-in: checklist

Lock-in is not evil; unmeasured lock-in is. Use this checklist to evaluate LLM vendor lock-in before a single provider owns your prompts, telemetry, and on-call muscle memory.

Related: OpenRouter vs direct OpenAI, comparison checklist tool, document prompt changes.

What “lock-in” means here

Not “we use Vendor A.” It means switching cost is high enough that product quality, legal posture, or margin would suffer for a quarter or more if Vendor A changed price, policy, or uptime.

Checklist (score each 0–2)

A. Interface & prompts

• System prompts avoid vendor-only markup where a portable subset works
• Tool / function schemas stored as your JSON, not only in a vendor UI
• Output contracts are schema-validated in your code (JSON schema prompts)
• Prompt versions live in git with owners

B. SDKs & infrastructure

• Thin adapter over HTTP; business logic not imported from vendor helpers
• Model ids mapped through an internal enum (Model.Small, Model.Strong)
• Timeouts, retries, and budgets are yours (backoff playbook)
• Secrets rotatable without redeploying prompt text

C. Quality & evals

• Golden sets run against at least two model families periodically
• Pass/fail thresholds documented; not “vibes from last demo”
• Latency and cost SLOs separate from a single SKU

D. Data & legal

• Training-use / retention terms reviewed for prod traffic
• DPA / subprocessors listed; exit includes data deletion timeline
• PII redaction happens before the vendor hop when required

E. Commercial

• Price change clause or at least a monitored alert on unit cost
• Spend caps and anomaly alerts (cost control)
• Documented fallback model with known quality delta

F. Exit drill

• Written “48-hour failover” playbook (who flips the adapter)
• Last drill date + issues found
• Customer comms template if quality drops during failover

Score guide: 0 = absent, 1 = partial, 2 = proven. Below ~18/30 on a production feature = schedule remediation before scaling spend.

Patterns that reduce lock-in without heroics

1. Router with two backends for the hottest path (even 10% can teach you)
2. Gateway only when terms & margins work — see pricing worksheet
3. Eval-gated promotion — new SKU must beat baseline on your suite
4. Config-as-code for prompts — switching models is a diff, not an archaeology dig

Anti-patterns

• Building product features inside a vendor’s proprietary agent canvas with no export
• Hard-coding tokenizer assumptions from one lab (local vs vendor tokenizer)
• “We’ll migrate later” with no drill and no second API key in staging

When deep lock-in is acceptable

• Regulated workflow certified on one stack
• Unique modality only one vendor offers and you accepted the risk in writing
• Prototype week — then re-score before GA

Print or copy this list into comparison checklist when vendor shopping. Revisit quarterly or after any pricing email.