From Prototype to Production

Online evaluation runs in production on live traffic. Offline sets go stale and never fully capture reality, so you also measure what actually happens: thumbs up/down, edit and acceptance rates, task completion, escalations to humans. Online signals catch the failures your offline set never imagined.

1. 1Curate an offline eval setCollect real inputs, edge cases, and past failures. Define what 'good' means for each. This is your regression test, run it on every prompt or model change.
2. 2Automate scoring where possibleDeterministic checks for format and required fields; LLM-as-judge for open-ended quality; a human-reviewed subset for the rest. Track a single quality number over time.
3. 3Instrument online signalsLog acceptance, edits, thumbs, completion, and escalation against each request. These reveal real-world quality your offline set missed.
4. 4Feed failures backEvery production failure becomes a new offline eval case, so the same mistake cannot ship twice. The eval set grows with the product.

Guardrails and error handling

Between the model and the user sits a layer that assumes the model can misbehave. Validate every output you depend on, if you asked for JSON matching a schema, parse and check it before use, and retry or fall back when it fails. Constrain inputs to guard against prompt injection and abuse, especially where user text is mixed with your instructions or feeds a tool call. Handle provider failures with timeouts, retries with backoff, and a defined behavior when the model is simply unavailable. The rule of thumb: never let unvalidated model output flow straight into something that renders to a user, executes an action, or hits your database.

Observability: logging, monitoring, tracing

When something goes wrong in production, and it will, you need to answer "what happened?" in minutes, not days.

Log enough to reconstruct any single request end to end, trace multi-step flows so you can see where time and failures accumulate, and monitor aggregates with alerts so a slow quality slide or a cost spike pages you before a user complains. Be deliberate about privacy: user inputs may contain sensitive data, so redact or restrict access to logs accordingly.

Cost controls and rate limits

An AI feature can spend money in a runaway loop faster than any traditional bug. Put ceilings in place before launch, not after the invoice:

• Per-user and per-key rate limits so one client (or one bad actor) cannot exhaust capacity or budget.
• Spend budgets and alerts at the account and feature level, with automatic throttling or shutoff at a hard cap.
• Loop and token guards on agents, maximum steps, maximum tokens, timeouts, so a stuck reasoning loop cannot bill indefinitely.
• Per-request cost logging so you can see unit economics and catch the expensive outliers.

Versioning that lets you change the model safely

Models are deprecated, repriced, and improved on the provider's schedule, not yours. If your product silently depends on "whatever the latest model is," every provider change is an uncontrolled experiment on your users. Instead: pin explicit model versions, and treat the prompt as versioned code alongside them. Roll out changes gradually, canary a new model or prompt on a slice of traffic, compare it against the current version on your evals and online signals, and keep the ability to roll back instantly. Because output varies run to run, a change that looks fine in a quick test can still regress in aggregate; the eval set and gradual rollout are what catch it.

Production-readiness checklist

Before you call an AI feature production-ready, you should be able to check every box:

1. 1EvaluationAn offline eval set that gates changes, plus online signals (acceptance, edits, escalations) captured in production.
2. 2GuardrailsOutputs validated before use; inputs constrained against injection; timeouts, retries, and a defined fallback for provider failures.
3. 3ObservabilityPer-request logging (with model/prompt version), tracing across the pipeline, and dashboards with alerts on quality, latency, cost, and error rates.
4. 4Cost controlsRate limits per user/key, spend budgets with alerts and a hard cap, and step/token guards on any agent loop.
5. 5VersioningPinned model versions, prompts versioned as code, gradual rollout with a canary, and a tested one-click rollback.
6. 6Human oversightConfirmation before costly or irreversible actions, and a working handoff path for cases the system should not resolve alone.

Further reading

• OpenAI, Production best practices
• OpenAI, Evaluating model performance (evals)
• Anthropic, Create strong empirical evaluations
• Anthropic, Reduce hallucinations and strengthen guardrails
• Sculley et al., Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015)