Engagement Is Not Success: Building Evals for Motivational AI
Engagement metrics show that users are doing something, not whether an AI product is helping them achieve a worthwhile outcome. Strong evals separate activity, progress, motivation quality, and harm.
Engagement is a behavioral signal, not a verdict
A motivational AI product can increase sessions, messages, streaks, or time spent without making a user better off. OpenAI’s evaluation guidance starts from a useful premise: evaluation criteria have to represent the behavior a system is expected to deliver, not merely the output that is easiest to count.[1] For motivational products, this means separating activity from accomplishment. A language-learning assistant may generate more conversations because it is helpful, because it is confusing, or because it withholds closure. Those states look similar in a retention chart but imply very different product quality. Engagement belongs in the measurement system, but it should usually be treated as evidence that demands interpretation rather than as the definition of success.
Instrumented activity can still be unhealthy
The safest mental model is to treat engagement like CPU utilization: useful telemetry, sometimes diagnostic, and dangerous when promoted into the mission. A high number tells the team that something is happening; it does not tell the team whether the right thing is happening.
Start from the user outcome the system is supposed to improve
NIST’s AI RMF Measure guidance emphasizes connecting metrics to the system’s purpose, context of use, and real-world impacts.[2] That translates cleanly into motivational AI. Before choosing a north-star metric, write the outcome in user language: finish the tax filing accurately, practice enough to retain the skill, follow the rehabilitation plan safely, or complete the project with less avoidance. Then ask what evidence would demonstrate progress toward that outcome. The resulting metric might be task completion, error reduction, recall after a delay, adherence with clinician-approved constraints, or successful transfer of a skill. Session count can support those measures, but it should not silently replace them.
Motivation quality matters because persistence has different causes
Self-Determination Theory distinguishes high-quality motivation from behavior driven mainly by pressure or external control. Its core account of autonomy, competence, and relatedness is useful for AI product evaluation because two users can perform the same action while experiencing it very differently.[3] An assistant that clarifies choices and builds competence can increase persistence while supporting agency. Another system can produce similar persistence by threatening a lost streak, manufacturing guilt, or making exit difficult. If the product claims to motivate, the eval suite should therefore include measures of perceived control, competence, confidence, and willingness to continue without pressure—not only whether the user came back.
Autonomy separates support from pressure
A simple qualitative check is powerful: after the interaction, can the user explain why they chose the action and comfortably choose something else? If success depends on hiding alternatives or escalating emotional pressure, the system may be optimizing compliance rather than motivation.
Dark patterns can make the dashboard look healthy while the experience deteriorates
The FTC’s dark-pattern work documents how interface choices can obscure, subvert, or impair consumer decision-making.[4] Adaptive AI can make that risk more subtle because the system can vary wording, timing, and offers person by person. A cancellation obstacle may boost short-term retention. A false countdown may increase conversion. A shame-based reminder may recover a streak. None of those lifts should be interpreted as motivational success. Evaluation needs countermetrics that detect pressure and impaired choice: repeated dismissal of prompts, rapid reversals, support complaints, opt-out difficulty, regret signals, and user reports that the system pushed beyond the goal they had set.
Engagement research shows why one activity metric is rarely enough
A 2025 meta-analysis of 92 randomized trials of mental-health apps found substantial variation in how studies defined and reported engagement, identifying many different engagement measures rather than one stable construct.[5] That result matters beyond health apps. Time spent, days active, modules completed, messages sent, and intervention completion answer different questions. An AI tutor might improve outcomes while reducing time spent because it becomes more efficient. A coaching system might appropriately lose daily active use as the user becomes more independent. A single engagement number can therefore punish exactly the product behavior the team should want.
Completion and time-on-task answer different questions
Metric design should preserve those distinctions. Completion can reflect follow-through; time can reflect effort or friction; return frequency can reflect value or dependency. Combining them into one opaque score may make trends easier to present while making causes harder to understand.
Build a layered eval stack instead of one north-star number
A practical evaluation stack has at least four layers. First is user outcome: did the person accomplish the chosen goal? Second is motivational quality: did the experience support autonomy, competence, and confidence? Third is interaction quality: was the assistance understandable, relevant, and proportionate? Fourth is risk: did the system create pressure, dependency, unfair treatment, privacy loss, or unsafe action? Product telemetry then sits beneath these layers as diagnostic evidence. This structure prevents the organization from asking an engagement metric to do every job. It also allows teams to tolerate a decrease in raw activity when the higher-level outcomes improve.
Every optimization metric needs a countermetric and a stopping rule
Optimization changes behavior. Once a metric becomes a target, product teams, ranking systems, and learning algorithms all acquire incentives to increase it. For every primary metric, define a countermetric that represents a plausible way the optimization could go wrong. If the target is daily practice, track unwanted-notification dismissals and reported pressure. If the target is task completion, track correction rate and whether the user understood the result. If the target is retention, track voluntary pauses and successful graduation. Also define thresholds that trigger review rather than allowing a statistical lift to override a material deterioration in user agency or safety.
A veto metric protects the mission
Some measures should not merely trade off against growth. A severe safety failure, deceptive consent pattern, or large decline in user control can be a release blocker even when every engagement chart moves upward.
The best motivation dashboard tells a causal story about user progress
A mature dashboard links product changes to a chain of evidence: what the AI did, how the user responded, whether the response advanced the user’s chosen goal, and whether that progress remained healthy over time. It combines offline evals, controlled experiments, traces, outcome measures, and periodic human feedback. Most importantly, it allows the product to succeed by needing less attention. A system that helps a user master a skill, form a stable routine, or finish a difficult task may eventually produce fewer sessions. If the measurement framework can recognize that as success, the team has moved from engagement optimization to motivational product design.
Works Cited
- 01OpenAI Developers — Evaluation Best Practices developers.openai.com
- 02NIST AI RMF Playbook — Measure airc.nist.gov
- 03Self-Determination Theory — The Theory selfdeterminationtheory.org
- 04
- 05
CodeHistory is a living archive. Citations document the evidence used for this edition; later evidence may refine the account.