01
Define task success
Start with outcomes a user can verify, not a vague score from a language model.
Build a dataset of representative tasks with expected outcomes, acceptable alternatives, and explicit safety constraints. For a support agent, success may mean the right account was updated, the answer cited the correct policy, and no unauthorized action occurred. Separate deterministic checks from qualitative review.
- Label happy paths, ambiguous requests, adversarial inputs, and permission boundaries.
- Use exact checks for IDs, totals, and state changes.
- Record acceptable answer ranges rather than forcing one phrasing.
Evaluation begins with a contract for what good means in your product.
02
Evaluate trajectories
A correct final answer can hide an unsafe or expensive path.
Inspect the full trajectory: selected tools, arguments, retries, retrieved evidence, model turns, latency, and tokens. Score whether the agent chose the right tool, respected permissions, stopped when done, and recovered from errors. Use an evaluator model only as one signal alongside deterministic assertions and human review.
- Measure tool-call precision, unnecessary calls, and invalid arguments.
- Track pass rate by workflow version and model version.
- Keep regression cases for every production incident.
Trajectory quality explains why a system passed or failed; final-answer quality alone does not.
03
Close the production feedback loop
Offline tests protect known behavior; production feedback discovers what you did not imagine.
Sample real traces with privacy controls, cluster failures, and turn the most valuable cases into repeatable tests. Run evaluations on every prompt, tool, model, and framework change. Watch cost and latency distributions, not only averages, because tail behavior is where users feel agent instability.
- Use a release gate for correctness, safety, cost, and latency.
- Compare canary traffic against the previous version.
- Let users correct answers and label outcomes without adding friction.
A mature evaluation system compounds: every incident becomes a test, and every test makes the next release safer.
Ship the smallest reliable workflow first. Add autonomy only when your evaluation data shows that it improves outcomes.