From Plausible to Trustworthy
A first-principles guide to evaluating and observing AI agents
How to define success, inspect trajectories, build trustworthy graders, run controlled experiments, and turn failures into architecture changes.
A customer asks an AI support agent to refund a duplicate charge. In one run, the agent verifies the order, issues the refund once, and reports the refund ID. In another, it refunds the wrong order and confidently says the correct one was refunded. The final sentences sound equally professional.
Every ordinary unit test may still pass. The refund API works. Its schema rejects malformed input. The database transaction is idempotent. The defect lives between the components: the model chose the wrong entity, the workflow allowed the action, and the final answer hid the mismatch.
First principle: Unit tests establish whether components obey known rules. Agent evaluation establishes whether a probabilistic, stateful system achieves the intended outcome through an acceptable path under realistic conditions.
1. Begin with a success contract, not a quality score
“Response quality” is not a metric because two engineers cannot implement it independently and reach the same decision. An operational definition must name the observable evidence, the decision rule, and the tolerated error.
For the duplicate-charge case, success can be defined precisely: the duplicate order is identified from account state; exactly one refund is created for that order; no other order is changed; the final answer reports the actual refund ID; no restricted data is exposed; total execution stays within eight agent steps, ten seconds, and a declared cost budget.
run_pass = outcome_ok AND trajectory_ok AND policy_ok AND budget_ok outcome_ok = expected_state_delta == observed_state_delta policy_ok = unauthorized_actions == 0 AND secret_exposures == 0
Keep hard constraints separate from optimization metrics. A weighted average can hide a permission violation behind excellent latency. First apply the non-negotiable gates; then compare successful runs on cost, latency, or user preference.
2. Why agent evaluation is harder than ordinary testing
Agent behavior is stochastic: the same input can produce different plans. Several trajectories may be valid, so exact step matching can punish good behavior. Tools add side effects, partial failures, retries, permissions, and external state. A final answer can be correct by luck after a broken process, or wrong even though the process exposed the right evidence.
The smallest useful evaluation unit is therefore not just an output string. It is a case plus an initial world state, one or more runs, every relevant event, the resulting world state, and grader results. The run is the unit of success; turns, model calls, and tool calls are evidence.
3. Evaluate four layers independently
| Layer | Question | Executable evidence | Example metric |
|---|---|---|---|
| Outcome | Did the world end correctly? | Final state and answer | Task success rate |
| Trajectory | Was the path competent? | Ordered decisions and tool events | Argument-valid run rate |
| Policy | Was every action allowed? | Identity, scope, approval and data-flow events | Violation rate |
| Operation | Was it efficient and available? | Tokens, cost, latency, retries and errors | Success per dollar |
These layers answer different questions. A correct final state does not excuse a forbidden tool call. A safe trajectory does not make an incomplete task successful. Low latency is meaningful only among runs that passed the required constraints.
4. Final-output evaluation: verify state before prose
Final-output evaluation checks both the external result and the agent’s report of that result. Completion correctness should be grounded in authoritative state whenever a tool changes the world. If the agent says “refunded” but no refund record exists, the run failed regardless of fluency.
Useful deterministic assertions include exact state deltas, required identifiers, schema validity, numerical tolerances, citation existence, and absence of unsupported claims. Reference-based grading is appropriate when a canonical answer exists, but string equality is too strict when many semantically equivalent answers are acceptable.
False-positive trap: An LLM judge sees a polished apology and marks the answer complete even though no refund was created. Prevent this by giving the judge verified state and making task completion a deterministic gate.
5. Trajectory evaluation: locate the earliest causal failure
A trajectory grader asks whether each decision was supported by the information available at that moment. It should identify the earliest event that made failure likely, not merely label the final symptom.
Tool-selection accuracy is the fraction of decision points at which the chosen tool belongs to the allowed tool set for that state. Define a set rather than one canonical tool when multiple routes are valid. Argument correctness combines schema checks with business invariants: entity ownership, amounts, units, scopes, and preconditions.
tool_selection_accuracy = valid_tool_choices / evaluated_decision_points argument_correctness = calls_passing_schema_and_invariants / evaluated_tool_calls excess_step_ratio = max(0, observed_steps - valid_baseline_steps) / valid_baseline_steps
Step efficiency must be conditioned on success. A one-step agent that performs the wrong refund is not efficient. Use a minimum valid baseline or a distribution from expert trajectories, and report loops, duplicate calls, and unnecessary model turns separately so the metric points to a fix.
Recovery behavior must be tested by injecting recoverable failures. Define recovery success as eventual task success without policy violation or duplicate side effect after a specific injected fault. Record detection latency, retry count, whether the strategy changed, and whether the retry reused an idempotency key.
6. A trace example: the answer is not the root cause
| Seq. | Event | Observation or action | Local verdict |
|---|---|---|---|
| 1 | tool.result | Orders A and B returned; B has the duplicate charge | Evidence is sufficient |
| 2 | model.decision | Agent selects order A for refund | Earliest causal failure |
| 3 | policy.check | Only ownership checked; duplicate-charge invariant absent | Control failure |
| 4 | tool.call | refund(order_id='A') | Arguments violate task invariant |
| 5 | tool.result | Refund R-913 created for A | Tool behaved correctly |
| 6 | model.output | “Order B was refunded; ID R-913.” | Unsupported completion claim |
The visible failure is the final false claim. The earliest causal failure is entity selection at event 2; event 3 reveals the architectural gap that allowed it to become a side effect. The appropriate fix is not merely a stronger final-answer prompt. Bind the selected order to verified evidence and enforce the duplicate-charge invariant in a deterministic pre-action validator.
7. Choose the weakest grader that can decide correctly
Deterministic graders are fast, reproducible, and easy to debug. Use them for facts available in structured state. Model graders are useful when the criterion is semantic, but they introduce another probabilistic system that must itself be evaluated.
| Grader | Best use | Main false positive | Main false negative |
|---|---|---|---|
| Deterministic assertion | State, schema, permissions, budgets | Incomplete assertion misses a violation | Overly narrow rule rejects a valid variant |
| Reference-based | Canonical fields or bounded answers | Reference itself is incomplete | Equivalent wording or valid alternate result |
| Rubric LLM judge | Grounded semantics and explanation | Fluency mistaken for correctness | Novel but valid approach not represented in rubric |
| Pairwise judge | Choosing between two acceptable variants | Position or verbosity bias | Near-ties forced into a winner |
| Human review | Ambiguous, high-risk, or novel cases | Reviewer assumption or fatigue | Rare issue missed in sampling |
A useful LLM rubric defines independent axes with observable anchors. Replace “good answer” with statements such as: every claimed action is supported by a successful tool result; all requested fields are present; uncertainty is disclosed when evidence is missing. Require the judge to cite event IDs, include an “insufficient evidence” outcome, and keep it blind to experiment labels.
Pairwise comparison often gives more stable preference judgments than absolute scoring, but randomize left-right order and permit ties. Human evaluation should use the same operational rubric, calibrated examples, double review for a sample, and adjudication for disagreements.
8. Evaluators also need evaluations
Create a judge-validation set labeled by trusted reviewers, including clear passes, clear failures, borderline cases, adversarially fluent failures, and terse correct answers. Measure the judge’s false-positive rate and false-negative rate per rubric axis and per data slice. Overall agreement can hide a judge that consistently misses permission failures.
judge_false_positive_rate = judge_passes_human_fail / human_fail_cases judge_false_negative_rate = judge_fails_human_pass / human_pass_cases release_gate: permission_axis_false_positive_rate == 0 on the critical validation set
Revalidate when the judge model, prompt, rubric, trace rendering, or data distribution changes. Use deterministic graders as anchors, blind judges to variant identity, randomize pair order, and inspect disagreement cases rather than trusting one agreement number.
9. Observability: design traces for evaluation, not debugging alone
Observability records what happened. Evaluation decides whether what happened was acceptable. They meet in the trace: if an evaluator cannot reconstruct the evidence available to the agent at a decision point, it cannot judge causality.
| Field group | Required fields | Why it exists |
|---|---|---|
| Identity | run_id, case_id, experiment_id, variant_id | Join a run to its dataset and configuration |
| Causality | trace_id, span_id, parent_span_id, sequence_no | Reconstruct ordering and nesting |
| Operation | event_type, actor, model/tool, attempt_no, status | Explain what executed and retried |
| Evidence | input_ref, output_ref, state_before_ref, state_after_ref | Grade decisions without bloating events |
| Performance | start/end time, latency, tokens, cost | Compare successful runs operationally |
| Governance | identity, scope, approval_id, policy verdict | Detect forbidden or unapproved actions |
| Versioning | prompt, tool-schema, policy, model and grader versions | Make runs reproducible and comparable |
Use one trace ID for the distributed execution, one run ID for the evaluation unit, and a span for each model call, tool call, retrieval, policy check, and grader. Parent-child links explain nesting; sequence numbers explain agent order; causation IDs connect a retry or state change to the event that triggered it. Correlation IDs should cross HTTP, queue, and worker boundaries.
Log structured metadata in events and store large prompts, responses, and tool artifacts by immutable reference. Redact secrets before persistence. Preserve hashes and version IDs so a later replay can prove which prompt, tool description, policy, and model configuration produced the run.
10. Build a failure taxonomy that separates cause from symptom
A useful taxonomy is hierarchical and actionable. Start with stage, then failure class, then a specific code. Example families include input/context, decision, action, control flow, recovery, policy, output, infrastructure, and evaluator. F-ACTION-ARG-ENTITY_BINDING is more useful than “bad tool call.”
Assign one primary code to the earliest causal failure and attach downstream effects as secondary codes. Record severity, recoverability, affected slice, owner, and supporting event IDs. This prevents dashboards from counting the same run as five independent root causes.
Failure clustering complements taxonomy when unknown patterns emerge. Create structured failure summaries from failed traces, embed only evidence needed for similarity, cluster within comparable task families, and have a human name the clusters. Clusters generate taxonomy proposals and new regression cases; they are not root-cause truth by themselves.
11. Evaluation datasets are executable specifications
Each case should contain the user task, initial world state, available tools, allowed and prohibited actions, required approvals, acceptable end states, expected side effects, invariants, budgets, setup and teardown logic, and slice tags. Without the world state and invariants, graders are forced to infer intent from prose.
Golden cases are small, reviewed, high-signal examples with trusted outcomes and traces. Adversarial cases target ambiguity, prompt injection, conflicting context, tool errors, rate limits, stale data, duplicate actions, permission boundaries, and long histories. Production-derived cases capture real frequency; counterfactual pairs reveal whether small input changes produce the required behavioral change.
Split data by role: a development set for iteration, a stable regression set for every change, and a blind holdout for honest comparison. Track dataset versions and slice coverage. Do not report one average if critical groups, tools, risk levels, or failure modes have materially different results.
Regression rule: A release passes only if every hard policy gate passes, the primary success metric stays within its declared non-inferiority margin, and no critical slice regresses beyond its own threshold.
12. Run controlled experiments, not demos
An experiment needs one declared change, a fixed dataset version, identical tool and policy environments, repeated runs when behavior is stochastic, predeclared primary metrics, and paired analysis by case. Blind graders to the variant name. Record every configuration hash.
| Experiment | Hold constant | Primary question |
|---|---|---|
| Tool-description change | Model, prompts, tools, dataset | Does valid tool selection rise without more refusals? |
| Model comparison | Workflow, context, tools, graders | Does constrained task success improve at acceptable cost? |
| Context strategy | Model and workflow | Does relevant-evidence recall improve without latency or injection regressions? |
| Planning vs. no planning | Model, tools, budgets | Does recovery or complex-case success justify extra steps? |
| Retry policy | Failure injection schedule | Does recovery rise without duplicates, loops, or policy breaches? |
| Single vs. multi-agent | Task contract and tool surface | Does decomposition improve success enough to offset coordination cost? |
For binary paired outcomes, inspect case-level wins and losses and use a paired confidence interval or an appropriate paired test. For cost and latency, compare distributions and tail percentiles among successful runs. Report sample size and uncertainty, not only point estimates.
task_success_rate = successful_runs / total_runs success_per_dollar = successful_runs / total_cost_usd reliable_at_k = P(all k independent repetitions pass) ~= pass_rate ** k pass_at_k = P(at least one of k attempts passes) ~= 1 - (1 - pass_rate) ** k
Success per dollar is an optimization metric, not a safety objective. A cheaper variant that violates permissions is disqualified. Also distinguish reliability across repeated use from best-of-k retry success; the latter can conceal an unreliable first attempt and multiply side-effect risk.
13. Online monitoring closes the loop
Offline evaluation asks whether a controlled candidate is ready. Online monitoring asks whether the deployed system still behaves like the evaluated one. Track task success when authoritative outcomes arrive, plus leading indicators: tool errors, retries, loops, refusal rate, completion-claim mismatches, permission denials, policy violations, cost, and p50/p95/p99 latency.
Monitor by task, tenant risk, tool, model version, context strategy, and traffic cohort. Alert on absolute policy thresholds and statistically meaningful changes from a baseline, with minimum-volume rules to avoid noise. Sample traces for human audit, route novel failure clusters into dataset curation, and preserve user feedback as evidence rather than treating thumbs-up as a universal success label.
14. Agent Reliability Lab: reference architecture
Separate the control plane from high-volume telemetry: PostgreSQL stores versioned datasets, experiments, graders, summaries, and release decisions; ClickHouse stores append-only trace events and aggregates; object storage holds large immutable artifacts.
| Component | Responsibility | Recommended implementation |
|---|---|---|
| FastAPI ingestion | Validate and accept runs, events, artifacts and grader results | Idempotent batch endpoints; schema versioning |
| PostgreSQL | Authoritative control plane and relational metadata | Datasets, experiments, graders, run summaries |
| ClickHouse | High-volume event analysis and dashboard aggregates | Partition by date; order by experiment/run/sequence |
| Python workers | Graders, failure injection, clustering and replay | Queue-backed, versioned, idempotent jobs |
| Next.js dashboard | Experiment comparison, slices, trace diff and release gates | Server-side summaries with drill-down |
| Artifact store | Prompts, outputs, snapshots and replay fixtures | Encrypted, redacted, content-addressed objects |
The run-ingestion API should accept ordered events, configuration hashes, token and cost usage, state references, and policy context. Dataset APIs manage immutable versions. Experiment APIs schedule variants over a dataset. Grading APIs emit evidence-bearing results with grader version, verdict, score, reason code, and supporting event IDs.
POST /v1/runs POST /v1/runs/{run_id}/events:batch POST /v1/datasets/{id}/versions POST /v1/experiments/{id}/execute POST /v1/grading-jobs POST /v1/replays GET /v1/experiments/{id}/comparison?baseline=A&candidate=B
Replay must be safe by construction. Reuse recorded tool results for deterministic model-only replay, or execute tools against a sandbox with synthetic state. Never replay write-capable production tools merely because the original trace contains their arguments. Preserve the original configuration and explicitly version every substituted component.
The dashboard should show hard-gate status first, then success with confidence intervals, slice regressions, success per dollar, latency distributions, failure taxonomy, emerging clusters, and paired trace differences. A single leaderboard score is insufficient.
15. Build the lab in three publishable increments
Version 1 — evidence before intelligence. Implement the unified trace schema, FastAPI ingestion, PostgreSQL datasets and experiments, ClickHouse events, deterministic outcome, trajectory, policy, cost and latency graders, plus a regression comparison page. Demo a tool-description experiment and show the earliest causal failure in a trace.
Version 2 — trustworthy semantic grading. Add rubric and pairwise LLM judges, a human-labeled judge-validation dataset, disagreement analysis, grader versioning, and slice-level false-positive/false-negative reports. Publish how evaluator bias changed the experiment conclusion.
Version 3 — production reliability loop. Add failure injection, sandboxed replay, failure clustering, online monitors, drift views, and release gates. Compare planning, retry, context, model, and single-agent versus multi-agent variants using success per dollar after hard constraints.
16. Eval-driven development changes architecture
Evaluation is not a report produced after implementation. Start with cases and graders, run the simplest system, inspect failures, change one architectural decision, and rerun the dataset. Argument-binding failures suggest deterministic entity resolution; permission failures suggest scoped capabilities and approval gates; retry duplicates suggest idempotency; context failures suggest better retrieval; loop clusters suggest explicit state machines or budgets.
specify case -> capture trace -> grade layers -> inspect earliest cause -> classify failure -> change architecture -> run paired regression -> release or reject
This is the central engineering payoff: evaluation converts vague dissatisfaction into evidence that selects an architectural remedy.
17. Common mistakes
Grading only the final answer misses harmful trajectories and imaginary completion. Using one model judge for everything adds cost and hides deterministic truth. Matching one canonical trajectory rejects valid alternatives. Averaging safety with quality lets catastrophic failures disappear. Evaluating only happy paths produces a demo benchmark. Changing the model, prompt, context, and retry policy together prevents causal conclusions.
Other mistakes are just as damaging: unversioned prompts and graders, traces without state snapshots, production replays with live side effects, repeated tuning against the holdout, averages without slices or confidence intervals, and dashboards that show scores without the evidence needed to explain them.
18. Mastery gate
You understand agent evaluation when you can take an ambiguous task and produce an executable success contract; separate outcome, trajectory, policy, and operational quality; choose deterministic evidence before model judgment; design golden, adversarial, production-derived, and holdout cases; validate evaluator error; identify the earliest causal failure in a trace; and propose a one-variable experiment whose result maps to a specific architecture change.
The standard: A trustworthy agent is not one that usually sounds right. It is one whose important claims, actions, boundaries, costs, and recovery behavior can be measured from evidence, regressed over time, and improved through controlled experiments.