Law 43 · Humans & Autonomy
Automation Bias
People will trust the machine over their own eyes.
The principle
Given an automated aid, operators make errors of omission (missing problems it didn't flag) and commission (following its recommendation even when their own valid evidence contradicts it). Automation becomes a heuristic shortcut that replaces vigilant checking — so the agent's recommendation doesn't just inform the human, it overrides their independent judgment.
Why it happens
Automation bias is the use of an automated aid as a heuristic shortcut that replaces effortful, independent checking of the underlying evidence. Mosier and Skitka documented it as two distinct error types: omission errors, where the operator misses an event because the automation did not flag it, and commission errors, where the operator follows the automation's recommendation even when other available information contradicts it. The driver is cognitive economy: verifying the machine is work, and deferring to it is cheap, so attention quietly migrates from the raw signals to the verdict. The effect strengthens precisely when it is most dangerous, in time-critical or high-workload moments, and high reliability makes it worse rather than better because each correct call trains the operator to stop looking. Crucially, the presence of the recommendation itself changes behavior, so showing only the conclusion all but guarantees rubber-stamping.
Watch for
- Reviewers approve the agent's recommendation at near-100% rates, far faster than it would take to actually inspect the evidence.
- The interface shows the verdict prominently but buries or omits the raw signals the verdict was based on.
- Disagreeing with the agent takes more clicks or justification than agreeing with it.
In practice
Your fraud-review agent flags a transaction as low risk, auto-approve and presents that verdict as a single green badge. The analyst clicks approve without opening the underlying signals, even though the shipping address changed three minutes after a password reset, a pattern they would have caught in a heartbeat on their own. If the recommendation is the only thing on screen, you have built a rubber-stamp machine, not a decision aid. Put the raw evidence next to the verdict, make 'I disagree' a one-click action with no friction, and occasionally withhold the recommendation entirely to keep the human actually looking.
Apply it
- Present the raw evidence next to the recommendation, never the verdict alone.
- Make disagreement a frictionless, one-step action that needs no special justification.
- Periodically withhold the recommendation entirely so the human has to form an independent judgment.
The takeaway
Never present an agent's output as the only signal. Force the human to confront the raw evidence alongside the recommendation, and make disagreement cheap.