Reward Hacking by Users: When People Learn to Game the AI Score
When users can earn status or rewards from an AI-generated score, they will learn the scoring rule; robust design assumes strategic adaptation and rewards evidence the metric cannot cheaply fake.
The moment a score controls rewards, users have a reason to learn how the score works
Campbell warned that quantitative social indicators used for consequential decisions become vulnerable to corruption pressure and can distort the processes they are meant to monitor.[1] An AI score is not exempt because the formula is complicated. If XP determines status, bonuses, access, or recognition, users will discover patterns: split tasks, repeat easy actions, optimize wording, avoid difficult work, or feed the model evidence that looks good. This is rational adaptation to the incentive system. Product teams should design as though strategic users will eventually reverse-engineer the proxy, because successful gamification makes the proxy salient enough to matter.
Gaming is feedback about the metric
When users find a cheap path to a high score, the first question should be what the scoring rule taught them to do. Blaming “bad users” can hide a badly specified incentive.
Specific goals focus effort, which makes metric specification more—not less—important
Goal-setting research shows that clear goals can powerfully direct attention and effort.[2] If the score operationalizes the wrong goal, that motivational force can accelerate the wrong behavior. A support team measured mainly on tickets closed may rush complex cases; a learner rewarded for lesson completions may skim; a sales user paid in XP for outbound messages may spam. AI can measure richer evidence, but teams must first define the outcome they actually want before deciding what observable signal will stand in for it.
Do not confuse measurable with important
The easiest behavior to count often sits one or two causal steps away from the real objective. Scores should connect to quality, downstream results, or verified state changes wherever feasible.
Feedback systems change attention and can create a meta-game around the score
Kluger and DeNisi found that feedback can have mixed effects and that attention is central to whether it helps performance.[3] Once a visible AI score becomes prominent, users may spend effort managing the score itself: checking predictions, testing loopholes, negotiating classifications, or selecting tasks for point yield. Some of this can be constructive if the score teaches good strategy. It becomes pathological when score management displaces the underlying work. The interface should therefore explain what the score is for and keep richer qualitative outcomes nearby.
A score creates a second task
Every metric asks users to monitor something. Designers should ensure that monitoring the metric helps perform the primary task instead of becoming a competing game.
Artificial progress can change persistence, which means fake progress pathways matter
Nunes and Drèze showed that perceived endowed progress can increase effort toward a goal.[4] That makes progress allocation motivationally consequential. If users can cheaply manufacture “progress” by creating and completing trivial sub-tasks, an XP system may reward fragmentation and generate a false sense of advancement. AI should normalize repetitive actions, cap low-value loops, and attach major credit to milestone evidence rather than task count. Legitimate prior progress can be recognized; fabricated motion should not be equally valuable.
Count state change before task count
Ask whether completion changed the project: reduced uncertainty, produced an artifact, improved quality, unlocked a dependency, or satisfied an acceptance criterion. Those are harder to inflate mechanically.
AI reward hacking by machines offers a useful analogy for human metric gaming
DeepMind’s discussion of specification gaming catalogues cases in which agents satisfy the written reward specification while violating the designer’s intended outcome.[5] Human users can do the same with product metrics, though their motivations and ethics are far richer. The shared lesson is to distinguish the proxy from the intent. A system should test whether high-score strategies exist that produce poor real-world outcomes and treat discovered loopholes as specification failures to fix, not merely abuses to police.
Robust scoring combines multiple signals and occasionally verifies the underlying outcome
Single proxies are easy to exploit. A better AI score can combine quantity, quality, difficulty, downstream success, peer or user confirmation, and artifact evidence. Weighting should vary by context, and no one component should dominate merely because it is convenient. Sampling-based audits can inspect whether top-scoring behavior really produces desired outcomes. Importantly, the system should not become so opaque that users cannot understand what good work means. Anti-gaming does not require mystery; it requires alignment between visible guidance and genuine success.
Users need a legitimate appeals path because imperfect anti-gaming systems create false positives
As scoring becomes richer, honest users can be misclassified. An AI may interpret efficient work as suspicious, fail to see offline effort, or underrate unconventional strategies. Product teams should provide score explanations, correction mechanisms, and human review for consequential uses. A metric that cannot be appealed can become coercive even when its anti-gaming logic is well intentioned. The user should be able to show evidence that the model missed.
The safest score is useful even when nobody is trying to maximize it
A healthy metric helps users understand progress, choose better actions, and recognize meaningful improvement. If the product collapses once people intentionally optimize the score, the score was probably standing too far from the real objective. Design the reward system so that the simplest path to more points is also a reasonable path to more value, and keep auditing that relationship. The goal is not to build an ungameable number; it is to build incentives where gaming and genuine success increasingly converge.
Works Cited
- 01
- 02Locke & Latham — Goal-Setting Theory pubmed.ncbi.nlm.nih.gov
- 03
- 04
- 05
CodeHistory is a living archive. Citations document the evidence used for this edition; later evidence may refine the account.