Law 42 · Humans & Autonomy
The Ironies of Automation
The more you automate, the harder the leftover human job becomes.
The principle
Automation doesn't shrink the human role — it transforms it into the hardest parts: passive monitoring plus rare, high-stakes intervention. Worse, by taking over the routine work, automation erodes the very skills and situational feel the operator needs when control is finally handed back. You design away the easy 95% and leave humans the 5% they're now least equipped to handle.
Why it happens
Bainbridge's core irony is structural: the designer hands the routine cases to the machine and leaves the operator only the situations the automation could not handle, which are by definition the hardest ones. Two reinforcing mechanisms make this worse over time. First, manual skill decays without practice, so the operator who must take over in an emergency is less competent than they were on day one. Second, situation awareness drops when you are passively monitoring rather than actively controlling, a pattern Endsley later named the automation conundrum: the more reliable and autonomous the system becomes, the less the supervising human understands its state and the harder it is for them to step back in. The leftover human role is not a smaller version of the old job; it is a different and more demanding one.
Watch for
- The human in the loop only ever sees the cases the agent already failed on, with no exposure to normal runs.
- When the agent escalates, it hands over a half-finished result with no explanation of what it tried or why it stopped.
- The people meant to supervise the agent can no longer do the task manually because the agent has done it for months.
In practice
You ship an invoice-processing agent that handles 95% of documents flawlessly, so the AP clerk now just watches a queue and approves the rare exceptions it kicks out. Six months later a malformed multi-currency invoice lands in their lap and they have no idea how to read it: they have not manually processed one since launch, and the agent gives them a half-finished extraction with no context on why it bailed. Do not dump the gnarly 5% on an operator whose skills you have quietly let atrophy. Keep them in the loop on a sample of normal cases too, and when you hand back, hand back the full reasoning trace and a clear statement of exactly what is stuck.
Apply it
- Route a sample of ordinary, successful cases to the human too, not just the exceptions, so their skill and context stay warm.
- On every handback, attach the full reasoning trace and a plain statement of exactly what is stuck and why.
- Design the escalation moment deliberately: make it rare, unambiguous, and accompanied by enough context to act on.
The takeaway
Don't just automate the happy path and dump edge cases on a human. Budget design effort for the residual role: keep the operator's context warm and make handback moments rare, clear, and well-supported.