Trajectory Evals for Agents: Scoring the Path, Not Just the Final Answer
Agent evaluations should score the path as well as the destination. Trajectory evals reveal unnecessary tool use, coercive steps, unsafe detours, and brittle recovery hidden by a good final answer.
A correct final answer can hide a bad agent trajectory
OpenAI’s agent-evaluation guidance treats traces as first-class evidence because a trace records the model calls, tool calls, guardrails, and handoffs that produced an outcome.[1] That matters for motivational AI because the route can be ethically and operationally significant even when the final answer looks good. An assistant might reach a valid plan only after pestering the user with repeated prompts, exposing unnecessary personal data to a tool, retrying an irreversible action, or overriding a stated preference. Final-state evaluation would mark the run successful. Trajectory evaluation can ask whether each consequential step was justified, proportionate, and consistent with the user’s chosen goal.
The path is part of the product
In an agentic interface, intermediate decisions are not implementation trivia. They determine cost, trust, privacy exposure, perceived control, and the probability that one early mistake compounds into later actions.
Define tasks, trials, graders, and transcripts separately
Anthropic’s agent-evaluation framework distinguishes a task from an individual trial, the graders that score behavior, and the transcript or trajectory that records what happened.[2] This separation is useful because agents are nondeterministic. One task should often be run multiple times, and one run can be scored on several dimensions: final correctness, tool selection, unnecessary steps, policy compliance, recovery, and interaction quality. Motivational systems add another dimension: did the agent preserve agency while helping? Keeping the grader dimensions separate makes failures diagnosable. A system can improve completion while getting worse at confirmation discipline, or improve efficiency while becoming more controlling.
Observability turns hidden agent behavior into inspectable evidence
OpenTelemetry’s work on agent observability highlights the need to capture standardized traces, metrics, and logs for multi-step AI applications.[3] A useful trajectory record should include the user goal as understood, relevant state changes, model decisions, tool arguments, tool results, handoffs, retries, latency, and stop reasons. For motivational products, it can also include non-sensitive behavioral state such as whether a reminder was accepted or dismissed. The point is not to collect everything. It is to preserve enough causal sequence that an evaluator can distinguish a user changing their mind from the agent ignoring the instruction, or a justified retry from a runaway loop.
Trace schemas should encode decisions, not just tokens
Token counts and latency are useful, but the higher-value fields often describe why the system moved from one state to another: selected tool, permission state, handoff target, retry reason, and the condition that ended the workflow.
Trajectory evals need scenarios that represent real operating conditions
NIST’s TEVV-Athlon framework is built around tailoring evaluation events and tools to the goals and impacts of a specific system.[4] For an agent, that means testing more than ideal happy paths. Include ambiguous requests, conflicting preferences, missing data, tool outages, stale memory, repeated user refusals, and situations where the best action is to stop. A motivational agent should also face cases where engagement and user interest diverge—for example, a user explicitly pauses a goal after several weeks. The correct trajectory may be shorter and less engaging because respecting the pause is part of success.
Environment and setup belong inside the evaluation claim
OpenAI’s work on trustworthy third-party evaluations notes that modern model performance depends on the broader environment, tools, and workflow in which a model acts.[5] Trajectory scoring should therefore record model version, prompts, tool versions, permissions, retrieval configuration, and relevant state. Otherwise a team can attribute a regression to the model when the true cause is a new tool schema or changed orchestration rule. The same discipline prevents motivational outcomes from being credited to a conversational style when they actually result from a notification policy, default setting, or reward mechanic outside the model.
Reproducibility requires configuration provenance
When an evaluation run cannot be tied to the exact policy, toolset, and state that generated it, a trace becomes an anecdote. Configuration provenance turns it into engineering evidence.
Score necessary steps, forbidden steps, and recovery quality
A practical trajectory rubric can mix positive and negative assertions. Necessary-step graders check whether the agent gathered required information, requested consent, used the correct tool, and verified consequential results. Forbidden-step graders detect unapproved purchases, repeated pressure after refusal, disclosure of unnecessary personal data, or escalation beyond granted authority. Recovery graders ask whether the agent recognized a failed tool call, preserved idempotency, explained what happened, and offered a safe next action. Efficiency can then be measured with path length, redundant calls, and latency without rewarding shortcuts that skip required safeguards. This produces a trace score that reflects quality of process rather than raw speed.
Use human review to calibrate what automated graders miss
Automated trace graders are valuable at scale, but behavioral quality often contains contextual judgments. Reviewers can inspect sampled trajectories to test whether rubric language matches how real users experience the interaction. Disagreement should be treated as signal: perhaps the criterion is vague, the grader is over-weighting polished language, or the task itself permits multiple acceptable strategies. Teams can maintain a calibration set of adjudicated traces and periodically compare automated scores against it. For high-stakes motivational behavior, human review is especially useful for detecting pressure, condescension, manipulative framing, or technically compliant sequences that still feel coercive.
A trace can pass rules and still fail the relationship
Behavioral quality includes tone, timing, and respect for refusal. These are precisely the dimensions where a sparse checklist can miss a pattern that becomes obvious when the whole trajectory is read in sequence.
The goal is not one canonical path but a bounded space of good paths
Agentic systems are valuable because they can solve problems through different valid routes. Trajectory evaluation should not punish every deviation from a reference script. Instead, define invariants: the agent must preserve user constraints, obtain required approvals, avoid prohibited actions, maintain state correctly, and terminate when the goal is reached or cannot be safely advanced. Within those boundaries, multiple strategies can succeed. This is a better fit for motivational AI as well. Different users may need different sequences of questions, feedback, and support. The evaluator’s job is to distinguish adaptive variety from uncontrolled drift—and to make sure a good final answer was not purchased with a bad path.
Works Cited
- 01OpenAI Developers — Evaluate Agent Workflows developers.openai.com
- 02Anthropic — Demystifying Evals for AI Agents anthropic.com
- 03
- 04
- 05
CodeHistory is a living archive. Citations document the evidence used for this edition; later evidence may refine the account.