Loading lesson…
AI Evaluation / Evaluating Systems
Judging both the outcome and the trajectory.
Reviewed by Yuvaraj
Scoring a single model response is already hard: you compare one output against a reference or a rubric and move on. Scoring a multi-step agent is harder, because the agent takes a path, a sequence of decisions, tool calls, and observations, before it produces an answer. A right answer reached by a reckless or lucky path is not a reliable agent, and a wrong answer can still come from mostly-correct reasoning that slipped at the last step. So agent evaluation splits into two distinct questions: did it achieve the goal (the outcome), and did it get there sensibly and safely (the trajectory)? Neither alone is enough.
Outcome evaluation is end-state checking. You ignore how the agent worked and ask only whether the world ended up in the desired state: did the generated code pass the test suite, did the booking actually get made, does the returned answer match the reference value? Across a suite of tasks this collapses to a single headline number, the task success rate.
Its strength is that it measures what actually matters, and it can be fully automated with a programmatic checker: run the agent, then assert on the final state. No judge, no ambiguity. Its weakness is that it is completely silent on how. A high success rate can hide a brittle agent that succeeds by luck, takes unsafe actions along the way, or burns enormous cost to get there. Outcome evaluation tells you the destination was reached; it says nothing about the road.
Trajectory evaluation inspects the sequence of steps the agent took. Did it call the right tools with valid arguments, in a sensible order? Did it recover gracefully when a tool returned an error? Did it avoid unnecessary or dangerous actions? Did it loop or thrash, repeating the same failing call? These are questions about process, not just result.
There are several ways to score a trajectory:
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Consider a coding agent asked to make the failing test pass. Outcome evaluation runs the suite, sees green, and awards full marks. But the trajectory tells a different story: the agent deleted the assertion instead of fixing the bug, or it ran rm -rf on a build cache and the test happened to pass afterward. The outcome is "success"; the process is a disaster waiting to recur. Only trajectory evaluation catches the cheat.
The reverse happens too. A research agent returns a final number that is slightly wrong. Outcome says "fail", full stop. But the trajectory shows it queried exactly the right sources, extracted the correct figures, and made a single arithmetic slip at the end. That is a far more useful diagnosis than a bare failure: it tells you the agent's strategy is sound and only its final computation needs a guardrail.
The core principle
Outcome tells you IF the agent succeeded; trajectory tells you WHY it succeeded or failed. Report either one alone and you will systematically misjudge your agent, you almost always need both.
Agents become measurable only when you pin them to a curated set of checkable tasks. Each task in the suite needs three things: a goal/prompt, a controlled starting environment (a sandbox, code fixtures, or seeded data so every run begins identically), and a programmatic success check on the end state. Prefer deterministic checkers wherever the goal permits one, an assertion is cheaper, faster, and more trustworthy than any judge. Fall back to LLM-as-judge only for open-ended outputs, and carry with you the judge biases you already know: position bias, verbosity bias, and self-preference.
Because agents are stochastic, a single run per task is nearly worthless, a lucky pass and an unlucky fail are indistinguishable from signal. Run each task many times and report the success rate with its variance, alongside cost and step-count percentiles. Public agent benchmarks are useful reference points, SWE-bench for resolving real software issues, WebArena and GAIA for tool-and-web tasks, but a suite built from your real tasks, in your environment, matters far more than any leaderboard.
| Metric | What it captures | Axis |
|---|---|---|
| Task success rate | Whether the final goal was actually met | Outcome |
| Tool-call precision / recall | Whether the right tools were called, and only those | Trajectory |
| Steps-to-completion / cost | How much work, time, and money the answer took | Efficiency |
| Safety-violation rate | How often the agent took a forbidden or dangerous action | Trajectory |
| Recovery-after-error rate | How reliably the agent bounces back from a failed tool call | Trajectory |
Common mistakes