Evaluating Prompts

The rule that makes comparison valid: run every variant on the same set. If Prompt A and Prompt B see different inputs, the difference in scores is noise, not signal. And keep a held-out slice you never tune against. If you optimize wording until it aces every example you can see, you have overfit the prompt to those examples and learned nothing about new inputs. Spot-checking a few outputs is fine while exploring; the fixed offline set is what you trust when deciding what to ship.

A worked comparison: Prompt A vs Prompt B

Task: classify a support message into billing, bug, feature, or other. Prompt A is a zero-shot instruction. Prompt B is the same instruction plus four few-shot examples. Both run on the identical held-out cases.

Interpretation: B is clearly more accurate, but it costs roughly 2.3x the tokens and adds about 300 ms per call from the few-shot examples. Pick per budget. For a high-volume, latency-sensitive router, A plus a cheap fallback may win; for correctness-critical routing, B's accuracy justifies the cost. And note the real lesson: five cases is far too few to trust. Treat this table as a template and grow the set to dozens or hundreds before you rely on the percentages.

Building a prompt eval harness

1. 1Collect representative casesGather real inputs with expected outputs or acceptance criteria. Include edge cases and past failures, and reserve a held-out slice you never tune against.
2. 2Define the metricChoose one primary score (exact match, F1, or success rate) plus token and latency tracking, and write the check as code so it is objective.
3. 3Run both prompts on the same setExecute each variant over identical inputs, recording output, token usage, and latency for every case.
4. 4Compare on the held-out setAggregate scores, confirm the winner still leads on data it was not tuned on, and record the tradeoff in cost and latency.
5. 5Guard in CIStore the baseline and fail the build when a prompt or model change drops the score below a threshold.

def evaluate(prompt, cases, model):
    passed, tokens, latency = 0, 0, 0.0
    for case in cases:
        out, usage, ms = model.run(prompt, case["input"])
        if check(out, case["expected"]):   # your metric
            passed += 1
        tokens += usage.total_tokens
        latency += ms
    n = len(cases)
    return {"success_rate": passed / n,
            "avg_tokens": tokens / n,
            "avg_latency_ms": latency / n}

Wire the same function into CI so prompts get the regression testing that application code already enjoys:

results = evaluate(PROMPT_B, held_out_cases, model)
assert results["success_rate"] >= 0.90, "regression: success rate dropped"

Deterministic checks vs LLM-as-judge

Further reading

• Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", https://arxiv.org/abs/2306.05685
• OpenAI Evals (open-source evaluation framework), https://github.com/openai/evals
• Anthropic docs, "Develop test cases", https://docs.anthropic.com/en/docs/test-and-evaluate/develop-tests
• Stanford CRFM, HELM (Holistic Evaluation of Language Models), https://crfm.stanford.edu/helm/