LLMOps Specifics

Non-determinism changes how you test

A classic model with fixed weights and a fixed input produces a fixed output, so tests can assert exact equality. LLMs sample. Temperature $T$ rescales the logits $z_i$ before the softmax,

$p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)},$

so higher $T$ flattens the distribution and increases output variance. Even at $T = 0$ (greedy argmax) you are not guaranteed byte-identical output across calls: floating-point non-associativity in batched GPU kernels, load-dependent batch composition, mixture-of-experts routing, and silent provider-side updates all introduce drift.

Regression testing therefore moves from exact match to property and distribution checks: assert that output is valid JSON against a schema, contains a required fact, or passes a rubric, and run each case $N$ times, tracking a pass rate $\hat{p} = k/N$ rather than a single result. A test that expects one canonical string will be flaky by construction.

Evaluation: golden sets and LLM-as-judge, not one accuracy number

Because outputs are free text, there is rarely a single accuracy number to report. Instead you maintain a golden set, a curated collection of representative inputs paired with expected answers or scoring rubrics, including edge cases and past failures, and score it along several axes: correctness, faithfulness/groundedness, format validity, safety, and tone.

Scoring uses whatever is cheapest and most reliable per axis: programmatic checks where possible (schema validation, regex, does the code compile), and LLM-as-judge for open-ended quality, where a strong model grades an output against a rubric or compares two outputs pairwise. Judges are scalable but biased, position bias, verbosity bias, and self-preference are well documented, so calibrate the judge against human labels before you trust its scores as a promotion gate.

Cost, tokens, and latency are first-class signals

Classic inference cost is roughly fixed per prediction; LLM cost is variable and dominated by token counts. Per request,

$\text{cost} = n_{\text{in}} \cdot p_{\text{in}} + n_{\text{out}} \cdot p_{\text{out}},$

where $n_{\text{in}}, n_{\text{out}}$ are input and output tokens and $p_{\text{in}}, p_{\text{out}}$ are the per-token prices for the chosen model. Cost balloons quietly through longer retrieved context, retries, and agent loops that call the model many times per task. Log tokens, cost, model id, prompt version, and latency, including time-to-first-token for streaming, on every request, and monitor p50/p95/p99 latency and cost per feature and per user against budgets.

Caching and its tradeoffs

Caching is one of the biggest cost and latency wins, and one of the easiest to get subtly wrong.

• Exact-match caching keys a stored response on a hash of the prompt plus parameters. Safe and fast for genuinely repeated calls, but it freezes one sample when $T > 0$, and every entry must be invalidated when the prompt or model version changes.
• Semantic caching embeds the query and returns a cached answer when a near-duplicate sits above a similarity threshold. It saves more but can serve a plausible-but-wrong answer to a query that was not actually equivalent, so the threshold is a correctness dial that needs monitoring.
• Provider prompt caching reuses a long static prefix (a big system prompt, shared instructions) server-side to cut input cost and latency on repeated prefixes. Structure prompts so stable content comes first; this reduces cost but does not cache the response.

Guardrails live in the serving path

A classifier can only emit a label; an LLM can emit anything, PII, toxic text, an injected instruction it was tricked into following, a hallucinated fact, or malformed output. Guardrails run inline as part of serving: input filters (prompt-injection detection, PII redaction), output filters (moderation, schema validation, groundedness checks against retrieved sources), and defined fallback behavior when a check fails. These filters add latency and cost, are often themselves model calls, and must be versioned and monitored for their own false-positive and false-negative rates.

Provider and model version changes

The weights are not yours, so the provider can update or deprecate a model and shift behavior, output style, format adherence, latency, and price, without you changing a line. Defend against silent drift: pin explicit dated snapshots rather than floating aliases like latest; treat a model bump like a dependency upgrade by re-running the full eval suite and canarying before rollout; watch deprecation notices; and keep a provider-abstraction layer so you can migrate models without rewriting the app.

Worked example: changing one prompt or one model version

You run a support-triage feature that classifies a ticket and drafts a reply. You want to edit one line of the system prompt. Here is everything that must be versioned and re-evaluated before it reaches users.

1. 1Bump the prompt versionCreate a new immutable version (v7 to v8) in the registry with a diff, author, and rationale. The old version stays deployable for instant rollback.
2. 2Re-run the golden setExecute every case in the pinned eval set against v8. Because outputs vary, run each case N times and record pass rates, not single outputs.
3. 3Score on every axisCompare correctness, JSON-schema validity, groundedness, safety, and tone against v7, plus the token/cost and latency deltas. Extra instructions usually add input tokens, so cost creeps up.
4. 4Gate on regressionsBlock promotion if any axis regresses past its threshold. A prompt that raises correctness but breaks format validity or triples cost is not a win.
5. 5Canary in productionRoute a small percentage of live traffic to v8 behind a flag and watch live cost, latency, guardrail trip rates, and user signals before widening.
6. 6Promote or roll backMove the flag to 100% or revert to v7, and either way log which prompt version served each request so a future regression is attributable.

Changing the model version runs the same pipeline, but the blast radius is larger: the model id is part of the artifact, so you re-baseline the entire eval suite rather than one case, re-check decoding parameters, re-price cost per token, re-validate every guardrail, and invalidate any cache entries keyed on the old model-plus-prompt pair.

How the two disciplines differ

The concrete checklist of what to version and what to log follows the same split:

Further reading

• MLOps: Continuous delivery and automation pipelines in machine learning, Google Cloud's reference for the classic-MLOps baseline this lesson contrasts against.
• Building LLM applications for production, Chip Huyen on prompt management, evaluation, and cost in LLM systems.
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Zheng et al., the canonical study of LLM-judge agreement with humans and its biases.
• OpenAI Evals, an open framework for building golden-set evaluations and regression suites for LLM outputs.
• OWASP Top 10 for Large Language Model Applications, the standard reference for prompt injection, output handling, and guardrail design.