Law 45 · Humans & Autonomy
Mind the Mode
Most automation surprises start with 'what mode is it in?'
The principle
Flexible, multi-mode automation produces 'automation surprises' — the system does something unexpected because the operator lost track of which mode it was in, what it would do next, and why. As autonomy grows, the human's job shifts to tracking its state, and every hidden mode transition becomes a latent failure path. An agent that silently changes how it behaves leaves its supervisor one step from being wrong about it.
Why it happens
Sarter and Woods traced automation surprises to a specific breakdown: a loss of mode awareness, where the operator no longer tracks the system's current and future status and behavior. The mechanism is a mismatch between the operator's mental model of what the automation will do and what it actually does, combined with an interface that fails to make state transitions salient. Their study of airline crews found these surprises cluster in non-normal and time-critical situations and often follow uncommanded or silent mode changes that the automation never announced. For an agent, every hidden shift, from planning to acting, from one tool policy to another, is a latent mode transition that leaves the supervisor reasoning about a system that no longer exists. The supervisor is then one step away from being wrong about everything the agent does next, not because the agent misbehaved but because its state became invisible.
Watch for
- The agent changes behavior, such as switching from drafting to executing, without surfacing that the switch happened.
- A supervisor cannot answer what mode is it in and what will it do next from the current display.
- Post-incident reviews repeatedly conclude I thought it was still just proposing.
In practice
Your coding agent silently switches from plan mode to auto-apply edits after a tool result, and the developer, still thinking it is drafting a proposal, watches it rewrite twelve files and run a migration. The surprise is not that it acted, it is that nobody knew which mode it was in or what it would do next. An agent that changes how it behaves without announcing it leaves its supervisor one step from being wrong about it. Render the current mode, the active guardrails, and the next intended action somewhere always visible, and make every mode transition an explicit, loud event the human has to see.
Apply it
- Keep the current mode, active constraints, and next intended action continuously visible.
- Make every mode transition an explicit, loud event the supervisor must see, never a silent switch.
- Treat any uncommanded change in behavior as a defect to surface, not an optimization to hide.
The takeaway
Make the agent's current mode, active constraints, and next intended action continuously visible — and never let it switch mode silently. Loud, legible state beats a clever agent the human can't predict.