Law 47 · Trust & Coordination
Trust Is Calibrated, Not Granted
Autonomy is earned in proportion to track record.
The principle
People extend an agent freedom the way they extend it to a new hire — incrementally, on reversible things first, widening the leash only as it proves itself. Both failure modes are real: over-trust causes misuse, under-trust causes a good capability to be abandoned. Reliance tracks the perceived reliability the system reveals, not just its true reliability.
Why it happens
Lee and See model reliance as a function of trust, and the central design goal is calibration: making perceived trustworthiness match actual reliability, so people rely on the system where it is strong and not where it is weak. Miscalibration runs both ways and both are costly. Over-trust produces misuse, where people lean on the system past its competence, while under-trust produces disuse, where a genuinely capable aid is abandoned. The under-trust failure has a sharp empirical edge: Dietvorst and colleagues showed algorithm aversion, where people lose confidence in an algorithm faster than in a human after seeing it make the very same error, and will then choose an inferior human forecaster instead. Because reliance tracks the reliability the system reveals rather than its true reliability, calibration is achieved by exposing where the agent is confident and where it is guessing, not by hiding its limits.
Watch for
- The agent is given broad write access to high-stakes systems before it has a track record on reversible ones.
- Every single action is funneled through manual approval, and the team is quietly abandoning the tool from fatigue.
- The agent presents strong and shaky outputs with identical confidence, giving users no basis to calibrate.
In practice
Two failure modes, both expensive. On day one you give the agent direct write access to production billing and it confidently double-applies a discount rule across 800 accounts. Or, burned by that, you wire every single action through manual approval, the team drowns in confirmation fatigue, and within a month they have quietly stopped using a genuinely capable tool. Calibrate instead of swinging between extremes: start it on reversible, low-stakes actions, widen the leash as its track record proves out, and surface where it is reliable versus where it is guessing so people lean on it exactly where they should and not an inch further.
Apply it
- Start the agent on low-stakes, reversible actions and widen its blast radius only as reliability is proven.
- Surface where the agent is reliable versus where it is guessing so users rely on it exactly that far.
- Avoid both extremes: neither hand it production write access on day one nor gate every trivial action behind approval.
The takeaway
Start the agent on low-stakes, reversible actions and expand its blast radius as reliability is proven. Show why it's confident where it's strong and flag where it's weak, so users lean on it exactly where they should.