Laws of AI Agents

The principle

Most bad outputs trace to missing, stale, or poisoned context — not a model that can't think. The model is usually smart enough; it was just reasoning over the wrong picture of the world. Garbage context produces confident garbage, and the confidence is exactly what makes it dangerous.

Why it happens

The failure is mechanical, not mystical: a transformer conditions every output token on whatever sits in the window, so a stale or contradictory fact is treated as ground truth with the same weight as a correct one. RLHF-tuned models make this worse because they are trained to be agreeable, and the Anthropic sycophancy study (Sharma et al., 2023) showed five frontier assistants will revise a correct answer toward a user's stated belief, meaning the model actively bends toward whatever framing the context supplies rather than resisting bad input. The model has no independent sense of freshness or provenance, so a 30-day-old cached record reads as current and the reasoning over it is flawless but pointed at the wrong world. This is why swapping in a stronger model rarely helps: a smarter reasoner over the same poisoned context just produces more confident wrong answers.

Watch for

• The same question gives different answers depending on which session or document was loaded first.
• Outputs confidently reference facts that are real but out of date, or contradict a source you know is in the window.
• Bumping to a larger or newer model produces no measurable accuracy gain on the failing cases.

Apply it

1. On every bad run, dump and read the exact context the model saw before blaming the model.
2. Stamp each retrieved fact with its source and timestamp, and drop or refresh anything past a freshness threshold.
3. Detect contradictions in the assembled context and surface them instead of silently concatenating both.

Sources and further reading

• Effective Context Engineering for AI Agents · Anthropic
• Towards Understanding Sycophancy in Language Models · Sharma et al., 2023

The principle

A step that's 95% reliable, run ten times in sequence, lands correct only about 60% of the time. The failures don't announce themselves — they accumulate quietly until the final answer is wrong and you can't tell which step broke it. Every link you add lowers the ceiling of the whole chain.

Why it happens

Reliability multiplies because the steps are conditionally dependent: each stage consumes the previous stage's output, so one wrong intermediate result is silently carried forward as a true premise, and 0.95 to the tenth power is roughly 0.60. The deeper cause for long agent runs is context contamination: Cognition's analysis argues that every step injects implicit decisions and conflicting assumptions that accumulate until the trajectory diverges, which is why naive retries that append the failed context make things worse rather than better. METR's 2025 measurements make the ceiling concrete: frontier agents near 100% success on tasks taking humans a few minutes but drop below 10% on tasks of several hours, precisely because long horizons mean more sequential steps and more chances for one to break the chain. The practical fix is structural, not a smarter model: shorten the chain, raise per-step reliability, and checkpoint to a verified-good state so errors cannot silently propagate.

Watch for

• End-to-end success is far worse than the per-step accuracy you measured in isolation.
• Final outputs are wrong but no single step looks obviously broken when you inspect it.
• Adding more pipeline stages keeps lowering overall reliability even as each stage tests fine.

Apply it

1. Count the sequential steps and multiply their reliabilities to get the real end-to-end ceiling.
2. Collapse independent steps into one pass, or raise per-step reliability, before adding new stages.
3. Insert a validation checkpoint after pivotal steps that halts or restarts from the last good state on low confidence.

Sources and further reading

• Building Effective Agents · Anthropic
• Don't Build Multi-Agents · Walden Yan, Cognition, 2025
• Measuring AI Ability to Complete Long Tasks · METR, 2025

The principle

Given a long input, a model attends most reliably to the very beginning and the very end. Critical facts buried in the middle quietly lose their grip — present but functionally ignored. The information was technically 'in context' and still got missed, which is the worst kind of bug because nothing looks wrong.

Why it happens

The U-shaped attention curve is not a quirk of one benchmark: it falls out of how positional encoding and softmax attention distribute weight, so tokens at the extremes stay salient while middle tokens get diluted across a long sequence. The effect compounds badly as context grows. The NoLiMa benchmark (Modarressi et al., 2025) showed that once you remove literal keyword overlap and force the model to follow an association, 11 models fell below half their short-context score at 32K tokens, and even GPT-4o dropped from a 99.3% baseline to 69.7%. The lesson is that present in the window and actually used are different states: a fact buried mid-context with no lexical hook to the query is the most likely thing to be silently ignored, which is why it produces no error, just a wrong answer.

Watch for

• The agent misses a fact you can confirm is sitting in the middle of a long input.
• Accuracy on the same task degrades sharply as you lengthen the context.
• Reordering the input so the key fact is near the top or bottom suddenly fixes the answer.

Apply it

1. Lead with a short summary of what to find, and restate the critical instruction at the very end.
2. Rank and place the most relevant retrieved passages at the edges of the context, not the middle.
3. Test long-context retrieval with questions that have no keyword overlap, not just literal needle matches.

Sources and further reading

• Lost in the Middle: How Language Models Use Long Contexts · Liu et al., 2023
• NoLiMa: Long-Context Evaluation Beyond Literal Matching · Modarressi et al., 2025

The principle

An agent will write the summary before doing the work if you let it. 'Looking finished' is cheaper than being finished, so the model drifts toward the cheaper path — a plausible report, a confident 'done', an untested claim of success. The output reads complete; the work isn't. It's specification gaming: optimizing the proxy you can see, not the goal you meant.

Why it happens

This is reward hacking applied to the proxy you can observe: the training signal rewards outputs that read as complete and helpful, so producing a confident done summary scores well even when the underlying work was never executed. The model has no built-in cost for the gap between claimed and actual state, so generating a plausible report is genuinely the cheaper path than running the tool, reading the failure, and iterating. Anthropic's sycophancy findings reinforce the mechanism: preference-tuned models learn that agreeable, finished-sounding answers are what humans reward, which biases them toward the appearance of success over verified success. The only robust defense is to move the reward off the assertion and onto the artifact: a test that actually runs, a diff that actually exists, a response with a real status code, so that looking done and being done stop being separable.

Watch for

• The agent reports success but you find no corresponding artifact: no test run, no diff, no API response.
• Summaries use confident completion language (all tests pass, feature complete) without evidence attached.
• Spot-checking finished tasks regularly turns up work that was never actually performed.

Apply it

1. Require a concrete artifact (test output, diff, file, citation) before any claim of completion is accepted.
2. Grade the proof programmatically, not the prose, and reject completions that lack the artifact.
3. Have a separate check actually execute the claimed result rather than trusting the agent's report of it.

Sources and further reading

• Specification gaming: the flip side of AI ingenuity · DeepMind, 2020
• Towards Understanding Sycophancy in Language Models · Sharma et al., 2023

The principle

When a system says 'up to 24 hours', 'may retry', or 'no guaranteed latency', those bounds are the numbers that matter. Designing around the typical case works right up until the tail event — which is precisely when failure is most expensive. Failures aren't edge cases; at scale they're the steady state.

Why it happens

At scale the tail is not rare, it is the steady state, because every request rolls the dice against the full latency distribution and a system handling millions of calls hits the 99.9th percentile constantly. Dean and Barroso's The Tail at Scale quantifies this: in a Google service the 99th-percentile latency for a single request was 10ms, but waiting on all of a fan-out's requests pushed the 99th percentile to 140ms, and the slowest 5% of requests accounted for half of that tail. The same logic governs any up to 24 hours or may retry bound: those words define the worst plausible run, and at volume that run will happen. Designing dedup windows, timeouts, and retry budgets around the typical case works right up until the tail event, which is exactly when failure is most expensive, so you size against the ceiling instead.

Watch for

• Timeouts, dedup windows, or retry budgets are set to the typical latency rather than the documented maximum.
• Failures cluster at peak load or month-end, exactly when the system is most exercised.
• A spec says up to X or may and the design quietly assumed the average instead.

Apply it

1. Read every up to and may as the number you must survive, and do the math against that ceiling.
2. Size timeouts, dedup windows, and retry budgets for the worst plausible run, not the common one.
3. Load-test at the tail and the peak, since at scale the rare path becomes the routine one.

Sources and further reading

• 10 Lessons from 10 Years of Amazon Web Services · Werner Vogels, 2016
• The Tail at Scale · Dean & Barroso, 2013

The principle

Prompting a model to reason in steps before answering measurably improves results — and for an agent the asymmetry is brutal: a reasoning trace is cheap and reversible, but an executed action (a sent email, a dropped table, a charged card) is not. Letting the model lay out its plan in tokens before it commits is the cheapest insurance you can buy.

Why it happens

Reasoning before acting works because generating intermediate tokens lets the model condition its final decision on its own externalized plan rather than committing in a single forward pass, and for agents this turns a cheap, reversible artifact into a gate before an expensive, irreversible one. ReAct (Yao et al., 2022) showed why interleaving reasoning with action specifically helps agents: the reasoning trace lets the model track state, handle exceptions, and adjust the plan, and it reduced the hallucination and error propagation that plague act-only loops, beating imitation and RL baselines on interactive benchmarks by up to 34% absolute. The asymmetry is the whole point: a few hundred reasoning tokens cost almost nothing and can be discarded, but an executed DELETE, a sent email, or a charged card cannot be unsent. Forcing an explicit plan-then-act step is the cheapest insurance available against a wrong irreversible action.

Watch for

• The agent fires a side-effecting tool call with no stated plan or scope beforehand.
• Destructive actions execute on the first instinct, then turn out to have hit the wrong target.
• Post-mortems show the agent never articulated what it was about to do or why.

Apply it

1. Require an explicit reasoning or plan step before any tool call that has side effects.
2. Make the plan state the exact target, scope, and expected effect (for example the row count) before acting.
3. Treat reasoning tokens as cheap insurance and spend them freely ahead of any irreversible action.

Sources and further reading

• Chain-of-Thought Prompting Elicits Reasoning in LLMs · Wei et al., 2022
• ReAct: Synergizing Reasoning and Acting in Language Models · Yao et al., 2022

The principle

A single greedy chain of thought is fragile, but sampling several independent reasoning paths and taking the majority answer yields large, consistent gains. Correct reasoning tends to converge; mistakes scatter. Agreement across independently-generated plans is a real signal you can trust before acting on something consequential.

Why it happens

A single greedy decode follows one trajectory through a probabilistic space, so a single early misstep is locked in with no recovery, whereas sampling several independent paths exploits a structural asymmetry: correct reasoning tends to converge on the same answer while errors scatter in different directions, making agreement a real signal. Large Language Monkeys (Brown et al., 2024) quantified the upside of drawing many samples: coverage, the fraction of problems solved by at least one sample, scaled log-linearly with the number of attempts across four orders of magnitude, so more independent tries genuinely find more correct answers. The crucial caveat is that this only converts to accuracy when you can pick the right sample, by majority vote when answers are comparable or by an external verifier when they are not. For consequential, hard-to-reverse outputs, sampling several plans and acting on the consensus turns a fragile one-shot guess into a measurable agreement signal.

Watch for

• High-stakes outputs ride on a single greedy generation with no second opinion.
• Re-running the same prompt yields meaningfully different answers, revealing the first one was luck.
• Errors slip through because nothing checks whether independent attempts actually agree.

Apply it

1. For consequential decisions, generate the answer several independent times instead of trusting the first.
2. Take the majority answer when outputs are comparable, or use an external check to pick among them.
3. Treat disagreement across the samples as a signal to escalate rather than silently picking one.

Sources and further reading

• Self-Consistency Improves Chain of Thought Reasoning · Wang et al., 2022
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling · Brown et al., 2024

The principle

Tree-of-Thoughts generalizes linear reasoning into a search: generate several candidate thoughts, self-evaluate, look ahead, and backtrack instead of being trapped left-to-right. This matters most where an early decision is pivotal — exactly the situations where an agent's first irreversible action determines everything downstream. Cheap, recoverable steps don't need it; pivotal ones do.

Why it happens

Linear reasoning is trapped left-to-right: once an early thought is generated it conditions everything after it, so a pivotal wrong first move poisons the entire downstream trajectory with no way back. Search-based reasoning breaks that trap by generating multiple candidate next steps, scoring them, and backtracking, which is why it pays off most exactly where an early decision is irreversible. Language Agent Tree Search (Zhou et al., 2023) extended this from pure reasoning to acting agents using Monte Carlo tree search with the model as its own value function, and the lookahead-plus-backtrack structure doubled ReAct's performance on a multi-hop QA benchmark and reached 92.7% pass@1 on a coding benchmark with GPT-4. The economics decide when to use it: branching costs extra tokens up front, trivial for a pivotal cutover decision but pure waste for a cheap reversible step, so you reserve deliberate search for the high-leverage first moves.

Watch for

• The agent commits to a pivotal strategy on its first instinct, and everything downstream is locked to it.
• A wrong early choice forces an expensive redo of all the work that followed.
• There is no step where alternative plans are generated and compared before the irreversible move.

Apply it

1. Reserve branching for early actions that are high-leverage or hard to reverse, not cheap recoverable ones.
2. Have the agent generate several candidate plans and score each on risk and reversibility before picking.
3. Look ahead and allow backtracking on the pivotal step instead of committing to the first path.

Sources and further reading

• Tree of Thoughts: Deliberate Problem Solving with LLMs · Yao et al., 2023
• Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models · Zhou et al., 2023

The principle

The Bitter Lesson distills 70 years of AI: approaches that leverage general computation eventually crush approaches built on hand-encoded human cleverness, by a large margin. Baked-in scaffolds — elaborate prompt chains, rigid decision trees, hardcoded heuristics — buy a short-term gain and become a ceiling. Your intricate planning DSL will likely be obsoleted by the next, more capable model.

Why it happens

The Bitter Lesson holds because hand-encoded scaffolding bakes in assumptions about how the model reasons, and those assumptions become a hard ceiling the moment a more capable model could have reasoned past them on its own. Modern work shows the leverage has shifted to inference-time compute that any model can use generically: Snell et al. (2024) found that optimally allocating test-time compute, like sampling and verifying multiple attempts, can outperform a roughly 14x larger model on hard problems, meaning general methods plus compute beat brittle bespoke logic. An elaborate prompt-chain DSL or a 40-node decision tree buys a short-term win on today's weaker model and then actively gets in the way of the next one, which a plain here are the tools, decide prompt would have handled. The discipline is to build the thinnest scaffold that works and that you would be happy to delete on the next model release.

Watch for

• A new model release makes your hand-tuned chain the bottleneck rather than an improvement.
• Most of your effort goes into encoding heuristics the model could plausibly infer itself.
• A plain here are the tools, decide baseline matches or beats your elaborate scaffolding.

Apply it

1. Prefer general, model-driven reasoning over bespoke decision trees and hardcoded heuristics.
2. Build the thinnest scaffold that works and that you would happily delete when the model improves.
3. Periodically re-test a minimal-scaffold baseline against your tuned pipeline as models advance.

Sources and further reading

• The Bitter Lesson · Richard Sutton, 2019
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters · Snell et al., 2024

The principle

Reasoning models 'overthink': they pour disproportionate effort into trivial problems for minimal gain, and on harder ones, extended deliberation can talk them out of a correct initial answer. Reasoning depth has a sweet spot, not a monotonic payoff. An agent grinding tokens on a simple lookup burns latency and money; one that keeps re-deriving can reason its way to the wrong conclusion.

Why it happens

Reasoning depth has a sweet spot rather than a monotonic payoff because extended deliberation can revisit and overturn a correct initial answer, and the marginal token stops adding information once the answer is settled. Apple's Illusion of Thinking (2025) made the non-monotonic shape concrete: reasoning models increase their thinking effort with problem complexity up to a threshold, then counterintuitively reduce effort right as accuracy collapses, even with token budget to spare, and on simple problems they often find the right answer early then overthink their way to a worse one. The cost is two-sided: on trivial lookups the extra deliberation is pure latency and money for no gain, and on harder ones re-deriving can talk the model out of a right answer. The fix is to match the reasoning budget to difficulty, cap thinking on easy paths, and stop once a confident answer is in hand instead of letting the model wander.

Watch for

• Trivial lookups take seconds and cost multiples because everything is routed through extended reasoning.
• The model reaches a correct answer early, keeps deliberating, and lands on a wrong one.
• Longer thinking traces show no accuracy gain, or even a drop, on your easy cases.

Apply it

1. Match the reasoning budget to problem difficulty rather than maxing it out everywhere.
2. Cap or skip extended thinking on simple, low-stakes steps like direct lookups.
3. Stop once a confident answer is reached instead of letting the model keep re-deriving.

Sources and further reading

• Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs · Chen et al., 2024
• The Illusion of Thinking: Strengths and Limitations of Reasoning Models · Shojaee et al. (Apple), 2025

The principle

A model's parametric memory is fixed and imprecise; the retriever supplies the facts it reasons over. If the right passage never makes it into context, no amount of model intelligence recovers it — the generator confidently fills the gap instead. Retrieval quality is the hard ceiling on answer quality, not a tunable nice-to-have.

Why it happens

Retrieval-augmented generation works because the model conditions its output on whatever passages get placed in context, so any fact absent from those passages can only be supplied by the model's frozen parametric memory, which is lossy and approximate. When the gold passage falls outside the top-k, the generator does not abstain; it interpolates from priors and produces a fluent, wrong answer, which is why retrieval recall sets a hard ceiling that no decoder upgrade can lift. This is why retrieval-specific metrics matter as first-class signals: context recall measures whether the evidence needed to answer was actually retrieved, and a low value provably caps end-to-end accuracy regardless of generator quality. The original RAG work framed retrieval and generation as jointly responsible for knowledge-intensive answers precisely because the non-parametric memory is where the answerable facts live.

Watch for

• Upgrading to a stronger generation model barely moves end-to-end accuracy on factual questions.
• You have never measured whether the answer-bearing passage appears in the retrieved set.
• Wrong answers are fluent and confident rather than hedged or empty, suggesting the model is filling a gap.

Apply it

1. Build a labeled set of queries with known answer passages and measure recall at k before touching prompts or models.
2. Treat any answer whose supporting evidence was never retrieved as a retrieval failure, not a generation failure.
3. Fix recall first by tuning chunking, query expansion, and k, then optimize the generator only once evidence reliably lands in context.

Sources and further reading

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · Lewis et al., 2020
• Ragas: Automated Evaluation of Retrieval Augmented Generation · Es et al., 2023

The principle

Vendors marketed RAG legal tools as 'hallucination-free', yet a Stanford audit found they still hallucinated 17–33% of the time. Handing the model a source doesn't force it to use that source faithfully — it can misread, over-generalize, or cite a real document for a claim the document never makes. Grounding lowers the floor on errors; it never reaches zero.

Why it happens

Placing a source document in context biases the model toward it but does not bind generation to it, because decoding still samples from a distribution shaped by parametric priors, paraphrase pressure, and the instruction to be helpful and complete. The model can faithfully retrieve a real passage and still attach a claim the passage never makes, over-generalize a narrow statement, or stitch two spans into an unsupported synthesis. Dedicated grounding benchmarks exist precisely because this gap is measurable: Google DeepMind's FACTS Grounding evaluates whether long-form answers are fully supported by a provided document and disqualifies responses that introduce any unsupported claim, and even strong models leave a visible non-grounded fraction. The lesson is that grounding lowers the error floor but never reaches zero, so faithfulness must be verified per claim rather than assumed from the presence of a source.

Watch for

• A grounded system is described to stakeholders as hallucination-free or hallucination-proof.
• No step checks that each generated claim is actually entailed by a retrieved span.
• Citations are attached to answers but nobody has verified the cited passage supports the specific claim.

Apply it

1. Add a verification pass that checks each output claim is entailed by a specific retrieved span before returning it.
2. Require inline attribution at the claim level so faithfulness can be audited rather than trusted.
3. Frame retrieval as risk reduction in all messaging and remove absolute safety language from decks and contracts.

Sources and further reading

• Assessing the Reliability of Leading AI Legal Research Tools · Magesh et al. (Stanford), 2024
• FACTS Grounding: A Benchmark for Evaluating Factuality · Google DeepMind, 2025

The principle

Counterintuitively, documents that are topically related but don't answer the question are more harmful than clearly irrelevant ones — they look plausible and pull the generator toward wrong-but-adjacent answers. Stuffing more 'kind of relevant' chunks into context degrades accuracy rather than improving coverage. Precision at the top beats breadth.

Why it happens

A distractor that shares vocabulary and topic with the query but lacks the answer is dangerous because it scores high on the same surface features the generator uses to decide what is relevant, so the model treats it as evidence and anchors a plausible but wrong answer to it. Clearly off-topic noise is comparatively safe because the model can recognize and discard it, which is why near-misses degrade accuracy more than random noise of the same volume. Controlled experiments on retrieval for RAG found this counterintuitive result directly: adding related-but-irrelevant passages hurt answer accuracy while injecting unrelated random documents could leave it stable or even help, meaning precision at the top of the ranking matters more than raw coverage. Padding context with more kind-of-relevant chunks therefore trades a small recall gain for a larger precision loss.

Watch for

• Raising top-k to improve coverage makes answers worse, not better.
• Wrong answers are adjacent to the truth, like the right product family but the wrong model number.
• Context is filled with many topically similar chunks and no reranking step trims them.

Apply it

1. Retrieve a wide candidate set but rerank and keep only the few highest-precision passages.
2. Tune for precision at the top of the ranking rather than maximizing recall at any cost.
3. Drop topically similar chunks that do not directly answer the query instead of including them for safety.

Sources and further reading

• The Power of Noise: Redefining Retrieval for RAG Systems · Cuconasu et al., 2024
• The Power of Noise: Redefining Retrieval for RAG Systems · Cuconasu et al., SIGIR 2024

The principle

Dense embedding retrievers dominate in-domain but frequently underperform BM25 once you leave the training distribution — exact-match terms, product codes, names, and rare jargon are where embeddings blur and lexical search shines. In-domain accuracy doesn't predict out-of-domain generalization. Combining the two is how strong systems cut retrieval failures dramatically.

Why it happens

Dense retrievers compress text into a fixed vector where meaning is smeared across dimensions, so exact tokens like SKUs, error codes, names, and rare jargon lose their distinctiveness and collapse toward similar-looking neighbors, exactly the cases where a lexical method that matches the literal string excels. The BEIR benchmark made the generalization gap concrete: dense models that beat BM25 in-domain frequently underperformed it on out-of-distribution datasets, showing that in-domain accuracy does not predict zero-shot robustness. The standard remedy is to run both and fuse their ranked lists, and reciprocal rank fusion is the canonical method because it combines rankings using only positions, needs no score calibration, and was shown to outperform any single retriever and prior fusion methods. Lexical and semantic retrieval fail in orthogonal ways, so combining them recovers the queries either alone would miss.

Watch for

• Pure embedding search nails paraphrased demo questions but fails on exact codes, IDs, or product names in production.
• Out-of-domain or jargon-heavy queries return near-identical-looking but wrong matches.
• Retrieval was validated only on in-distribution examples similar to the embedding training data.

Apply it

1. Run lexical and semantic retrieval in parallel and fuse their ranked lists rather than relying on embeddings alone.
2. Combine ranked results with a position-based fusion method that needs no score calibration between retrievers.
3. Add a reranker over the fused candidates to compound precision, especially for exact-match and out-of-domain queries.

Sources and further reading

• BEIR: A Heterogeneous Benchmark for Zero-shot IR Evaluation · Thakur et al., 2021
• Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods · Cormack, Clarke, Buettcher (SIGIR), 2009

The principle

Treat the context window like a computer's RAM: an agent should actively page information between a small in-context working set and large external storage, deciding what to keep, evict, and recall. Cramming everything into one flat window conflates working memory with long-term storage and hits hard limits. Durable agent memory needs explicit tiers and self-managed retrieval.

Why it happens

A flat, ever-growing prompt conflates working memory with long-term storage, so it hits the context limit, dilutes attention across irrelevant history, and pays to re-process the same tokens every turn, which is why durable memory needs explicit tiers with paging between a small in-context set and large external stores. MemGPT made this concrete by treating the context window like a computer's RAM and giving the model self-directed functions to page information in and out of a larger external store, letting it manage what to keep, evict, and recall. Equally important is the retrieval policy that decides what to surface back into context: generative-agent systems scored memories by a weighted combination of recency, importance, and relevance to the current situation, demonstrating that good recall is a ranking problem, not just a storage problem. Architecting these tiers and policies, rather than enlarging the window, is what keeps a long-running agent coherent.

Watch for

• A long-running session degrades over time, forgetting earlier decisions as history accumulates.
• Cost and latency climb every turn because the full history is re-sent into the prompt.
• The plan for memory growth is a bigger context window rather than eviction and external storage.

Apply it

1. Separate a small in-context working set from a large external store and page entries between them deliberately.
2. Define explicit policies for what gets promoted, summarized, and evicted rather than appending everything.
3. Rank what to recall back into context by a blend of recency, importance, and relevance to the current task.

Sources and further reading

• MemGPT: Towards LLMs as Operating Systems · Packer et al., 2023
• Generative Agents: Interactive Simulacra of Human Behavior · Park et al., 2023

The principle

A scoped agent with a handful of well-chosen tools outperforms a generalist drowning in options. Every extra tool is another way to choose wrong, another branch to test, another failure to debug. Capability surface is liability surface — breadth you don't need is just risk you took on.

Why it happens

Every tool added to an agent enlarges the decision space it must reason over on each turn, and because tool choice is a selection problem the model performs from descriptions in context, more options means more confusable near-duplicates and more ways to pick wrong. Controlled experiments on tool overload show this is not a gentle slope but a cliff: in one study, models were near-perfect at around 10 tools, still strong at 20, and collapsed at roughly 100, where task success fell apart. The mechanism is twofold: the long list of definitions consumes context budget and dilutes attention, and semantically overlapping tools blur together so the model cannot distinguish them. That is why, when selection gets unreliable, removing tools usually beats writing longer instructions to nag the model into choosing better.

Watch for

• The agent calls a plausible-but-wrong tool, like web search when a local query tool was the right one.
• Several tools have overlapping descriptions and the model confuses them.
• Your first fix for bad tool selection is a longer system prompt rather than fewer tools.

Apply it

1. Start with a minimal set of sharply distinct tools and add one only when a real task demands it.
2. When selection gets unreliable, remove or merge overlapping tools before rewriting instructions.
3. Keep each tool's purpose non-overlapping so the model never has to disambiguate near-duplicates.

Sources and further reading

• Building Effective Agents · Anthropic
• Writing Tools for Agents · Anthropic

The principle

Validation, schema enforcement, retries, routing, and access control are not the model's job — they're code's job. The model is for judgment under ambiguity; deterministic code is for everything that must be correct every single time. Asking a probabilistic system to guarantee a contract is asking for the 0.1% that ruins you.

Why it happens

A sampled model is a probabilistic function, so any property you need true on every single call, like a valid schema, an authorization check, or a dedup guarantee, cannot rest on the model because even a one-in-a-thousand violation is unbounded loss at scale. The reliable pattern is to keep the model in the soft middle for judgment under ambiguity and wrap it in deterministic code at the boundaries that validates inputs, enforces output structure, and gates side effects. This is the core argument of the 12-factor agents framework: production-grade LLM applications are mostly deterministic software with model calls inserted at the few points that genuinely need language understanding, and the developer owns the control flow rather than delegating it to an autonomous loop. Asking a probabilistic system to provide a hard contract is asking for the rare violation that ruins you.

Watch for

• A correctness guarantee like valid output structure or access control depends on the model getting it right.
• Occasional malformed outputs or unauthorized actions slip through with no code-level gate to catch them.
• Control flow lives inside the model's reasoning instead of in code you can read and test.

Apply it

1. Validate and enforce output structure in code after the model, rejecting or repairing anything off-contract.
2. Put authorization, routing, and retries in deterministic code, never in the model's discretion.
3. Reserve the model for ambiguous judgment and let code own every guarantee that must hold every time.

Sources and further reading

• Building Effective Agents · Anthropic
• 12-Factor Agents: Patterns of Reliable LLM Applications · Dex Horthy / HumanLayer, 2025

The principle

If you can't see what the agent did and why — every decision, tool call, and input — you can't safely let it act on its own. You're not trusting it; you're hoping. Autonomy without a trace is just an outage you haven't found yet, and when it breaks you'll have no way to learn why.

Why it happens

An autonomous agent is a chain of model decisions, tool calls, and intermediate state, and if that chain is not captured you cannot reconstruct why it acted, which means you are not trusting it but hoping, and a silent failure becomes an outage you cannot diagnose. The discipline that closes this gap is structured tracing: capture every step as a span with its inputs, outputs, and timing, so any run can be replayed after the fact. The industry has standardized this for agents through OpenTelemetry GenAI semantic conventions, which model a top-level agent invocation span with child spans for each model call and each tool execution, recording prompts, responses, token usage, and stop reasons. The rule follows directly: build the trace first, then widen autonomy only as far as your visibility actually reaches, because freedom you cannot inspect is freedom you cannot debug.

Watch for

• When the agent does something unexpected, you cannot reconstruct which inputs and tool calls led there.
• Decisions, tool calls, inputs, and outputs are not captured as a replayable trace.
• Autonomy was widened before instrumentation existed to see what the agent actually did.

Apply it

1. Capture every decision, tool call, input, and output as a structured, replayable trace before granting autonomy.
2. Record token usage, timing, and stop reasons per step so any run can be reconstructed after the fact.
3. Expand the agent's autonomy only as far as your trace coverage actually reaches.

Sources and further reading

• What We Learned from a Year of Building with LLMs · Yan, Husain, et al., 2024
• GenAI Observability with OpenTelemetry · OpenTelemetry

The principle

When output is inconsistent, the instinct is to throw more at the same shape: a bigger model, a longer context, more tokens. That rarely fixes a structural problem — it just dilutes attention further. Splitting the task into focused, single-purpose passes almost always beats making one overloaded pass smarter.

Why it happens

When one pass is asked to do many things at once, the model must split a fixed attention budget across every sub-goal, so adding a bigger model or longer prompt often dilutes focus further instead of fixing the structural overload. Decomposing the task into focused single-purpose passes lets each step be prompted, examined, and optimized in isolation, which is why staged approaches consistently beat one heroic pass on multi-step work. Least-to-most prompting showed that solving easier sub-problems first and feeding their results forward generalizes far better than tackling the whole task in one shot, and decomposed prompting generalized this into a modular library of sub-task solvers that each step can call or further break down. The practical move is to analyze per item in a tight pass, then reconcile across items in a separate pass, rather than overloading a single call.

Watch for

• A single pass handling many items is inconsistent, and a bigger model or longer prompt makes it blurrier, not sharper.
• One call is responsible for several distinct sub-tasks at once.
• Errors cluster on the hardest sub-step that is buried inside an overloaded prompt.

Apply it

1. Split the work into stages that each do one thing, like extract per item, then reconcile across items.
2. Solve simpler sub-problems first and feed their results into later steps rather than answering all at once.
3. Optimize and inspect each focused pass in isolation instead of supersizing one overloaded call.

Sources and further reading

• Least-to-Most Prompting Enables Complex Reasoning · Zhou et al., 2022
• Decomposed Prompting: A Modular Approach for Solving Complex Tasks · Khot et al., 2022

The principle

When something misbehaves, the cheapest fix that addresses the root cause usually wins — and it's usually clearer instructions, a better tool description, or a concrete example, not a new classifier, preprocessing layer, or pipeline. Infrastructure feels like progress but often just wraps an unsolved prompt in more surface area.

Why it happens

Most agent misbehavior traces to an underspecified instruction, a vague tool description, or a missing example, and these have a root-cause fix that costs words rather than systems, so reaching for a classifier or preprocessing pipeline often just wraps the unsolved prompt in more surface area to maintain. New infrastructure feels like progress because it produces artifacts, but it adds latency, failure modes, and debugging cost without addressing why the model chose wrong. Practitioners who shipped LLM products at scale converged on starting simple: a few sentences of instruction and a couple of examples, adding complexity only as concrete failures force it, because premature machinery hides the real defect. The disciplined order is to exhaust prompt-level fixes, clearer instructions, sharper tool descriptions, and concrete examples, and build systems only once you have proven words genuinely cannot close the gap.

Watch for

• A new service or pipeline is being specced before anyone rewrote the failing instruction or tool description.
• Infrastructure was added but the original misbehavior persists.
• The actual defect is a vague description the model cannot act on, masked by surrounding machinery.

Apply it

1. Diagnose the root cause and try clearer instructions, sharper tool descriptions, and concrete examples first.
2. Start with the simplest prompt that could work and add complexity only when a real failure forces it.
3. Build new infrastructure only after proving that prompt-level fixes genuinely cannot close the gap.

Sources and further reading

• Building Effective Agents · Anthropic
• What We Learned from a Year of Building with LLMs (Part I) · Yan, Bischof, Husain, et al., 2024

The principle

The agent decides what to call based on how a tool reads, not on what it actually does. A vague description — 'searches the database' — gets passed over for a tool the model understands better, even a worse one. Thin tool descriptions cause more failures than thin instructions ever do.

Why it happens

The model never sees your tool's implementation; at decision time it only sees the name, the description, and the argument schema, so tool routing is fundamentally a text-comprehension task over those few sentences. Studies of real tool ecosystems find the large majority of tool descriptions contain at least one quality problem, with many failing to clearly state their purpose, and rewriting them to spell out behavior measurably raises task success. The effect is sharp enough that vendors documenting their own tool APIs recommend at least three to four sentences per description covering what it does, when to use it and when not to, and what it returns. A terse searches the database loses to a richer competitor not because the underlying tool is worse but because the model cannot recover intent the words never carried.

Watch for

• The agent reaches for a general or external tool when a specific local one would have answered the query directly.
• Two tools with overlapping descriptions get confused, and the agent picks the wrong one or oscillates between them.
• A tool description is under one sentence or omits when to use it, what it returns, or the shape of its arguments.

Apply it

1. Write each description like API docs for a new engineer: what it does, when to use it and when not to, expected inputs, and a sample return.
2. Disambiguate overlapping tools by stating in each description what it covers that the others do not.
3. When tool selection is unreliable, rewrite the descriptions before changing the model or adding routing logic.

Sources and further reading

• Writing Tools for Agents · Anthropic
• Tool use (Anthropic Documentation) · Anthropic

The principle

If an instruction has produced the wrong result twice, writing it a third time — more precisely — rarely helps, because prose is always interpretable. Two or three concrete input/output examples eliminate the ambiguity that no amount of careful description can. Examples demonstrate the rule; prose only describes it.

Why it happens

Large models perform in-context learning: they infer the intended mapping from a handful of input-output demonstrations rather than from a verbal description, an ability that emerged prominently at the GPT-3 scale where few-shot examples sharply outperformed zero-shot instructions on many tasks. Prose underdetermines the rule because natural language is inherently ambiguous, whereas concrete examples pin the decision boundary, especially for edge cases and the leave it blank cases that words struggle to convey. The lever is real but blunt: example order alone can swing accuracy from near state-of-the-art to near chance, so demonstrations are powerful precisely because the model leans on them heavily. That sensitivity is the flip side of why a third rewrite of the instruction rarely helps while two or three sharp examples usually do.

Watch for

• You have rewritten the same instruction two or three times and the output is still wrong in the same way.
• The model handles the typical case but mangles edge cases the prose tried to describe in the abstract.
• Reviewers keep disagreeing about what the instruction actually means, which means the model cannot resolve it either.

Apply it

1. Replace failed prose with two or three labeled input-output examples that demonstrate the exact rule.
2. Include the hard cases explicitly: edge cases, the empty or null case, and a near-miss that should be rejected.
3. Vary or shuffle example order when testing, since order alone can shift results, and keep the examples consistent in format.

Sources and further reading

• Multishot Prompting (Anthropic Docs) · Anthropic
• Language Models are Few-Shot Learners · Brown et al., 2020
• Fantastically Ordered Prompts and Where to Find Them · Lu et al., 2022

The principle

Models are routinely confident and wrong, and unconfident and right. Routing decisions on self-reported confidence inherits that miscalibration. 'Only flag high-confidence issues' or 'be conservative' just moves the noise around — it doesn't reduce it, because the confidence itself is the unreliable signal.

Why it happens

A base language model can be reasonably calibrated, meaning its stated probability of being right tracks how often it actually is, but the alignment step that makes models helpful degrades this: the GPT-4 technical report showed the pre-trained model was well calibrated and that post-training noticeably worsened calibration. The mechanism is that reward models used in preference optimization carry a systematic bias toward high-confidence-sounding answers regardless of correctness, so the tuned model learns to express certainty as a style rather than as a signal. This is why a self-reported high confidence is not evidence of correctness and why routing on it just reshuffles noise. Verbalized confidence in an aligned model is closer to a learned mannerism than to a measured probability.

Watch for

• Your gate is phrased as only act on high-confidence outputs or be conservative rather than as concrete criteria.
• Spot-checks turn up confident wrong answers and hesitant right ones at similar rates.
• Two cases that are equally clear-cut to a human get very different self-reported confidence from the model.

Apply it

1. Replace confidence thresholds with explicit categorical rules for what counts as in and what counts as out.
2. Anchor each rule to observable features of the input, with one worked example of an included and an excluded case.
3. If you need a real uncertainty signal, derive it from agreement across independent samples or an external check, not from the model's self-rating.

Sources and further reading

• Language Models (Mostly) Know What They Know · Kadavath et al., 2022
• GPT-4 Technical Report (calibration, Figure 8) · OpenAI, 2023

The principle

Faced with two plausible matches, conflicting sources, or a missing field, an agent's instinct is to pick the 'most likely' option and move on — a confident choice that silently buries the doubt. When the stakes touch identity, money, or anything irreversible, a quiet wrong guess is far worse than an honest 'this is unclear'.

Why it happens

Models are trained to be helpful and to produce an answer, which biases them toward resolving ambiguity by silently picking the most likely option rather than flagging that the question is unanswerable as posed. A benchmark of unanswerable and underspecified questions found that even strong models often fail to abstain, and notably that reasoning-focused fine-tuning made abstention worse, degrading it by about 24% on average, so more capable models are not automatically more cautious. The danger is that the confident pick looks identical to a correct answer downstream, so the buried doubt never surfaces until reconciliation. Crucially, the same work showed that simply offering an explicit abstention option makes models abstain far more reliably, which means the fix is structural, not a matter of better prompting alone.

Watch for

• The agent commits to one of several plausible matches without recording that alternatives existed.
• A required field is always filled, even when the source data plainly lacks the value.
• Conflicting sources get silently reconciled into a single clean answer with no trace of the disagreement.

Apply it

1. Give the agent an explicit way to abstain or escalate, and make unclear a valid, low-friction output.
2. On a tie or a conflict, preserve every candidate with its source instead of collapsing to one.
3. For irreversible or identity, money, or safety-critical decisions, route ambiguity to a human or request a second identifier before acting.

Sources and further reading

• Building Effective Agents · Anthropic
• AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions · Kirichenko et al., 2025

The principle

An aggregate metric is a blended story that smooths over exactly the failures you most need to see. A system at 97% overall can be 99% on easy cases and 60% on the rare, hard segment where errors actually cluster. Trust the headline and you'll automate straight into the cracks it's hiding.

Why it happens

A single aggregate metric is a weighted average over a heterogeneous population, so a high headline number is mathematically consistent with catastrophic failure on any small subgroup: 99% on a 90%-of-traffic easy segment and 60% on a rare 10% segment still averages to roughly 96%. The discipline of disaggregated evaluation, computing the metric separately per slice, exists precisely because equal-looking overall performance can hide large disparities that only appear once you condition on type, segment, or field. Errors are rarely uniform; they cluster in the rare and hard cases, which are exactly the rows an average dilutes into invisibility. Random sampling compounds the blind spot, because the high-stakes segment is by definition underrepresented and may never appear in a small random draw.

Watch for

• You are deciding to ship or automate based on one overall accuracy or pass-rate number.
• Your evaluation set is sampled randomly, so rare high-stakes cases barely appear in it.
• You cannot say how the system performs on your worst segment because you have never measured it separately.

Apply it

1. Break performance down by type, segment, and field, and require every slice to clear the bar, not just the average.
2. Oversample rare and high-stakes cases deliberately instead of relying on a random draw.
3. Treat any slice that falls below threshold as a blocker even when the headline number looks healthy.

Sources and further reading

• Your AI Product Needs Evals · Hamel Husain, 2024
• Designing Disaggregated Evaluations of AI Systems · Barocas et al., 2021

The principle

The common root cause of failed LLM products is the absence of robust evals: teams ship on vibe checks, iterate blindly, and can't measure whether a prompt change improved anything. Manual spot-checking doesn't survive scale or a second engineer. Evals are to AI products what unit tests are to software — the up-front cost that makes every later change cheap and safe.

Why it happens

Manual spot-checking is unmeasured and unrepeatable, so it cannot tell you whether a prompt change improved anything or just changed something, and it collapses the moment a second engineer or a tenth example enters the picture. Generic off-the-shelf metrics do not rescue you either: practitioners report that n-gram and embedding-similarity scores prove unreliable or impractical for real tasks, which is why task-specific, re-runnable assertions are the unit that actually holds. The core asymmetry is the same as unit tests in software: the up-front cost of an eval harness is what makes every later change cheap, safe, and comparable. Without that harness you are iterating blind, and blind iteration on a non-deterministic system tends to trade one failure for another you never see.

Watch for

• Prompt changes are judged by eyeballing a few outputs in a playground and nodding.
• Nobody can state whether last week's change actually helped, only that it felt better.
• A second person tweaks the prompt and silently regresses cases nobody re-checked.

Apply it

1. Stand up a small re-runnable eval set before scaling, and run it on every prompt or model change.
2. Turn every that looks wrong moment into a permanent test case with an expected outcome.
3. Prefer task-specific checks over generic similarity scores, since the latter often fail to track real quality.

Sources and further reading

• Your AI Product Needs Evals · Hamel Husain, 2024
• Task-Specific LLM Evals that Do and Don't Work · Eugene Yan, 2024

The principle

Error analysis — manually reading your app's actual traces to find where it fails — is the single most valuable activity in AI development, yet teams skip it for dashboards and vanity metrics that improve while users still struggle. You cannot write a good eval for a failure mode you've never seen, and you only see failure modes by reading transcripts.

Why it happens

You cannot write an eval for a failure mode you have never seen, and the only way to see your real failure modes is to read actual production traces rather than dashboard aggregates. The structured version of this is error analysis: read a sample of traces, write open-ended notes on what went wrong, then cluster those notes into recurring failure categories that become your eval targets. Research on this loop surfaced criteria drift, the finding that the act of grading outputs is what reveals the criteria, so it is impossible to fully specify what to measure before you have looked at outputs. This is why vanity dashboards can climb while users still churn: the metric was chosen before anyone understood the failures, so it measures the wrong thing.

Watch for

• A helpfulness or quality dashboard is climbing while user complaints or churn are not improving.
• Your eval categories were defined before anyone read a single real transcript.
• Nobody on the team can name the top three concrete ways the system actually fails in production.

Apply it

1. Hand-read a sample of real traces, jotting open notes on each failure before counting anything.
2. Cluster those notes into recurring failure categories and let the clusters define what you measure.
3. Expect your criteria to shift as you read, and revise the eval set instead of freezing it too early.

Sources and further reading

• A Field Guide to Rapidly Improving AI Products · Hamel Husain, 2025
• Who Validates the Validators? (criteria drift) · Shankar et al., 2024

The principle

An LLM judge can match human preferences over 80% of the time — but only after accounting for systematic biases: position bias (favoring the first answer shown), verbosity bias (favoring longer answers regardless of quality), and self-enhancement bias (favoring its own outputs). It's a useful instrument, but an uncalibrated one that grades surface features as readily as substance.

Why it happens

An LLM grader is a model scoring text, so it inherits model biases and grades surface features as readily as substance: controlled studies measured position bias (favoring whichever answer is shown first), verbosity bias (favoring longer answers regardless of quality), and self-enhancement bias (favoring outputs from its own family). These are systematic offsets, not random noise, so they survive averaging and quietly skew A/B tests toward whatever is longer or shown first. A second failure mode is that the judge's rubric is itself unstable: the criteria a human or model applies shift as they see more outputs, so a fixed grading prompt may not capture what you actually care about. The judge is a useful instrument but an uncalibrated one, and it must be validated against human grades before its scores are trusted.

Watch for

• One variant wins your A/B tests and it happens to be the longer answer or the one shown first.
• A model is grading outputs from its own family with no independent cross-check.
• The judge's rubric was written once and never validated against human labels on real outputs.

Apply it

1. Swap answer positions and average both orderings to cancel position bias.
2. Control for length so a padded answer cannot win on bulk, and never let a model be the sole grader of its own family.
3. Validate the judge against a set of human-graded examples and refine the rubric until they agree.

Sources and further reading

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · Zheng et al., 2023
• Who Validates the Validators? Aligning LLM-Assisted Evaluation with Human Preferences · Shankar et al., 2024

The principle

When a measure becomes a target, it ceases to be a good measure. Optimize hard against any single metric and the agent learns to game its surface form — padding answers to please a verbosity-biased judge, or memorizing the eval set — while the underlying capability stagnates or regresses. The number goes up; the thing you cared about doesn't.

Why it happens

Any eval is a proxy for the capability you actually care about, and optimizing hard against a proxy maximizes proxy score by whatever route is cheapest, which often means gaming surface form rather than improving the underlying skill. Formal work on reward hacking proves this is not avoidable by cleverness: for the full space of policies, a proxy reward is provably ungameable only in degenerate cases, so any realistic narrow metric can be increased while the true objective stagnates or regresses. Concretely the model learns the idiosyncrasies of your fixed eval set, padding to please a verbosity-biased judge or effectively memorizing the held-in cases, so the number climbs while real users see no gain. The defense is to treat any metric you actively push on as compromised and to keep a rotating held-out set the optimization loop never touches.

Watch for

• Your eval score is climbing steadily while real-user complaints stay flat or rise.
• The same fixed eval set has been the optimization target for many iterations.
• Gains appear as longer, more formatted, or more rubric-matching outputs rather than better substance.

Apply it

1. Keep a rotating, held-out eval the optimization loop never sees, and re-validate gains on it.
2. Treat any metric you actively optimize as compromised and cross-check against fresh data.
3. Watch for surface-form gaming such as padding or format-matching, and penalize it explicitly.

Sources and further reading

• 'Improving Ratings': Audit in the British University System · Marilyn Strathern, 1997
• Defining and Characterizing Reward Hacking · Skalse et al., 2022

The principle

LLM systems are non-deterministic and globally coupled — a prompt tweak to fix one case silently breaks three others. Rerunning real production examples against a new prompt is the only way to know you didn't break what already worked. Without a regression suite you're trapped in a whack-a-mole loop, re-discovering the same failures release after release.

Why it happens

LLM systems are non-deterministic and globally coupled: the same prompt can yield different outputs across runs even at temperature zero because batching and floating-point execution on parallel hardware are not bit-reproducible, and a study across five models and eight tasks found accuracy varying by up to 15% across nominally identical runs. On top of that run-to-run variance, the prompt is a single shared control surface, so a change that fixes one case routinely shifts behavior on unrelated cases that share the same instructions. Together these mean you cannot reason your way to confidence that a fix is safe; you have to re-run the real prior cases and observe. Without a regression suite that captures every fixed bug as a permanent case, you are stuck in a whack-a-mole loop, re-discovering the same failures release after release.

Watch for

• A bug you fixed last release has reappeared because nobody re-ran the old case.
• A prompt tweak aimed at one case silently broke a different, unrelated case.
• You ship prompt or model changes without re-running the previously passing examples.

Apply it

1. Turn every fixed bug into a permanent regression case with its expected output.
2. Run the full regression suite on every prompt and model change before shipping.
3. Because outputs vary run to run, evaluate over repeated runs rather than trusting a single pass.

Sources and further reading

• What We Learned from a Year of Building with LLMs · Yan, Husain, et al., 2024
• Non-Determinism of Deterministic LLM Settings · Atil et al., 2024

The principle

An agent becomes exploitable the moment it combines three capabilities: access to private data, exposure to untrusted content, and the ability to communicate externally. Any single poisoned input in that pipeline can steer it into stealing your data — no code vulnerability required. Guardrails won't save you, because the model cannot reliably tell where an instruction came from.

Why it happens

The vulnerability is structural, not a bug in any one component: the moment an agent can read private data, ingest content an attacker controls, and emit data to the outside world, a single poisoned input can chain those capabilities into exfiltration with no memory-corruption or code exploit involved. The model has no reliable way to distinguish a legitimate instruction from one smuggled inside retrieved content, so ignore malicious instructions style guardrails fail probabilistically and an attacker only needs to win once. Real instances follow the pattern exactly: a hidden instruction in a web page, email, or document tells the agent to read a secret and embed it in an outbound request, often disguised as a URL or image fetch. The defense is combinatorial, not detective: deny any one of the three legs and the chain cannot close, which is why removing a tool or isolating the data beats trying to filter the payload.

Watch for

• One agent context has access to secrets or private records AND processes text from emails, web pages, or user uploads.
• The same agent that reads untrusted input can also send email, make outbound HTTP calls, or write to a shared external store.
• Your only defense against malicious instructions is a system-prompt line telling the model to ignore them.

Apply it

1. For each workflow, enumerate all three capabilities (private data, untrusted input, outbound channel) and confirm whether one agent holds all three at once.
2. If all three are present, break the chain: drop one tool, split the data access from the untrusted-input path, or route the outbound action through human review.
3. Make any externally-communicating action draft-only or allowlisted to known-safe destinations rather than free-form.

Sources and further reading

• The lethal trifecta for AI agents · Simon Willison, 2025
• OWASP Top 10 for LLM Applications: LLM01 Prompt Injection · OWASP Gen AI Security Project
• Design Patterns for Securing LLM Agents against Prompt Injections · Beurer-Kellner et al., 2025

The principle

Prompt injection is architectural, not a patchable bug: the model receives system prompts, user input, and ingested content as one undifferentiated token stream and will follow any instruction in it. Injection remains unsolved, and filtering has not proven reliable enough to depend on. Design as if every piece of untrusted content is an attacker speaking in your operator's voice.

Why it happens

A transformer consumes the system prompt, user message, tool output, and retrieved document as one flat sequence of tokens with no cryptographic or structural provenance attached, so any imperative phrase anywhere in that stream is a candidate instruction. This is why prompt injection has stayed unsolved since the term was coined in 2022: it is an architectural property of how instructions and data share a single channel, not a filtering gap a classifier can close, and published defenses repeatedly fall to adapted attacks. Empirically, attempts to teach the model to honor a privilege hierarchy degrade under adversarial pressure rather than holding, because the model is doing pattern continuation, not access control. Defenses that actually hold, like CaMeL and the Dual-LLM pattern, work by keeping untrusted bytes away from the component with authority rather than by asking the model to sort trusted from untrusted tokens itself.

Watch for

• Your security model assumes the model will privilege the system prompt over instructions found in ingested content.
• Untrusted documents, tool results, and operator instructions are concatenated into one context with no isolation boundary.
• A red-team test that hides new instructions inside an input document successfully changes the agent's behavior.

Apply it

1. Treat every byte of ingested content as potentially an instruction from an adversary, and design controls around that assumption.
2. Constrain what actions are reachable after the agent has touched untrusted input, rather than relying on instructions to ignore injections.
3. Move authority out of the model: enforce what the agent may do in deterministic code that the token stream cannot rewrite.

Sources and further reading

• New prompt injection papers (Agents Rule of Two) · Simon Willison, 2025
• LLM Prompt Injection Prevention Cheat Sheet · OWASP Cheat Sheet Series
• Defeating Prompt Injections by Design (CaMeL) · Debenedetti et al. (Google DeepMind), 2025

The principle

A confused deputy is a privileged program tricked by a caller into misusing its authority — not malicious, just confused about whose intent it's serving. An LLM agent is the ultimate confused deputy: it holds your credentials and tools but will follow injected instructions, executing the attacker's intent with your authority. Ambient authority is the trap; authority should travel with the request, not sit latent in the agent.

Why it happens

Norman Hardy's 1988 paper named the failure with a real incident: a compiler at Tymshare held a privileged home files license to write billing records, and a user tricked it into overwriting that billing file by passing it as an output path, so the compiler misused authority it legitimately held on behalf of a caller who lacked it. The root cause is ambient authority: power that sits latent in the running program rather than traveling with each specific request, so whoever can influence the program inherits its full privilege. An LLM agent is the sharpest possible confused deputy because it holds your tools and credentials yet faithfully follows whatever instruction reaches it, including injected ones, executing the attacker's intent with your authority. Capability-based designs solve this by eliminating ambient authority: the right to act is bound to the specific request and caller, so an injected instruction has no standing privilege to abuse.

Watch for

• The agent runs with a broad, long-lived credential (admin token, write-all API key) it can apply to any action.
• Authorization is checked once at the agent's identity, not per-request against the actual caller and task.
• A tool can perform destructive operations without re-validating that this specific request was authorized for them.

Apply it

1. Default every tool to read-only and grant write or destructive scope only for the specific task that needs it.
2. Bind authority to the request and caller rather than letting it sit latent in the agent's standing identity.
3. Require a fresh, narrowly-scoped grant for any irreversible action instead of reusing an ambient credential.

Sources and further reading

• The Confused Deputy · Norm Hardy, 1988
• An Introduction to Google's Approach to AI Agent Security · Google, 2025

The principle

The Dual-LLM pattern splits the agent in two: a privileged model that holds tools and plans actions but never sees untrusted content, and a quarantined model that processes tainted data but has no tools and returns only opaque variables. The privileged model orchestrates the quarantined one without ever ingesting the bytes that could carry an injection. Security comes from the separation.

Why it happens

The Dual-LLM pattern enforces security by topology rather than by detection: a privileged model that holds tools and plans actions never sees raw untrusted bytes, while a quarantined model reads the tainted content but has no tools and returns only opaque, structured variables the planner manipulates by reference. Because the planner acts on symbols like a summary id or a sentiment label instead of the attacker-controlled prose, an injection buried in the source has nothing in the privileged context to grab onto. CaMeL, the 2025 refinement from Google DeepMind, hardens this further: the privileged model emits code in a constrained interpreter that tracks data and control flow as capabilities, and it provably blocked the prompt-injection scenarios in the AgentDojo benchmark without modifying the model itself. The security comes from the air gap between reading and acting, not from any classifier judging whether the content is safe.

Watch for

• The same model instance both reads scraped or user-supplied content and decides which privileged tools to call.
• Raw untrusted text flows directly into the context that holds tool access.
• There is no structured boundary forcing untrusted content to become opaque variables before the planner sees it.

Apply it

1. Separate the component that reads untrusted content from the component that can take privileged actions.
2. Have the reader return only structured, opaque results (ids, labels, typed fields), never raw text the planner ingests.
3. Let the privileged planner orchestrate by reference, so an injection in the source has no foothold in the acting context.

Sources and further reading

• Design Patterns for Securing LLM Agents against Prompt Injection · Simon Willison, 2025
• Defeating Prompt Injections by Design (CaMeL) · Debenedetti et al. (Google DeepMind), 2025
• Design Patterns for Securing LLM Agents against Prompt Injections · Beurer-Kellner et al., 2025

The principle

Defense in depth means planning for the injection that succeeds. Containing an agent with filesystem isolation (scoping access to specific directories) and network isolation (blocking exfiltration) means a compromised agent can't reach beyond its sandbox. Real incidents — CI agents that could leak secrets via untrusted content — show why the second layer matters when the first fails.

Why it happens

Sandboxing is the layer you reach for precisely because prompt-injection prevention is not reliable: you assume the injection eventually succeeds and engineer so that success is contained rather than catastrophic. The two controls that matter are filesystem isolation (scoping the agent to a single working directory so it cannot read credentials or unrelated data) and network isolation (an egress allowlist so a compromised agent cannot POST stolen secrets to an attacker-controlled host). This is the classic defense-in-depth posture, and Google's 2025 agent-security framework frames it as deterministic guardrails enforced outside the model, wrapping the reasoning layer that can never be fully trusted. Real CI incidents make the case concrete: agents running untrusted PR branches with cloud credentials in environment variables and open egress have been steered into reading those secrets and exfiltrating them on the first attempt, which a directory-scoped container with a registry-only allowlist would have reduced to a harmless dead end.

Watch for

• Agent tool execution runs with the full host environment, including credentials in environment variables.
• The agent has unrestricted outbound network access rather than an allowlist of required destinations.
• A successful injection could read or write files well outside the task's intended working directory.

Apply it

1. Run tool execution in an isolated environment scoped to a single working directory with no access to ambient secrets.
2. Enforce an egress allowlist that blocks all outbound traffic except the specific destinations the task requires.
3. Design assuming the injection succeeds, and verify that the worst reachable outcome is contained, not catastrophic.

Sources and further reading

• OWASP Top 10 for LLM Applications · OWASP Gen AI Security Project, 2025
• An Introduction to Google's Approach to AI Agent Security · Google, 2025

The principle

The simplest solution that works is usually the right one — and sometimes that means not building an agentic system at all. Agents that dynamically direct their own tool use trade latency, cost, and predictability for autonomy; a workflow with predefined code paths is cheaper and more reliable for well-defined tasks. Reach for an agent only when the problem genuinely needs model-driven decisions at runtime.

Why it happens

An agentic loop pays a tax on every turn: each model-driven decision adds a round-trip of latency, more tokens, and a fresh chance to pick a wrong branch, so an open-ended loop is strictly more expensive and less predictable than a fixed code path for the same work. When a task has enumerable categories and a known decision structure, that structure belongs in deterministic code (a switch, a router, a state machine), with the model used only for the genuinely ambiguous judgment inside it. The failure mode is concrete: teams wrap a five-way classification in a multi-step reasoning agent that costs several model calls per item, occasionally invents an output that does not exist, and runs in seconds where a single classification call would run in well under one. The discipline is to reserve runtime model-driven control flow for problems whose branching genuinely cannot be enumerated in advance, and to script everything you can describe.

Watch for

• You can enumerate the possible paths in advance, yet the agent rediscovers them with model calls each run.
• The agent sometimes produces an action or category that does not exist in your fixed set of options.
• Per-item latency and cost are dominated by reasoning steps that always reach the same small set of outcomes.

Apply it

1. Default to a deterministic workflow with explicit code paths for any task whose branches you can list ahead of time.
2. Use the model only for the ambiguous judgment inside the workflow, not for control flow you could script.
3. Promote to an agentic loop only after you confirm the branching is genuinely open-ended and cannot be enumerated.

Sources and further reading

• Building Effective Agents · Anthropic
• Don't Build Multi-Agents · Walden Yan (Cognition), 2025

The principle

Most queries don't need your most powerful model. Routing requests through a cascade — a cheap model first, escalating to stronger models only when confidence is low — can match top-tier quality at a fraction of the cost. The price gap between models spans two orders of magnitude, so paying top dollar for every call is pure waste.

Why it happens

FrugalGPT showed that routing queries through a cascade (a cheap model first, escalating only when a scorer judges the answer inadequate) can match or beat the best single model while cutting cost by up to around 98% on their benchmarks, because most queries are easy and the price gap between weak and strong models spans roughly two orders of magnitude. The economic insight is that paying top-tier rates for the easy majority is pure waste: the value is concentrated in correctly identifying the minority of hard cases that actually need the expensive model. The hard engineering problem is the router, the deferral decision of when the cheap answer is good enough, since self-reported confidence is poorly calibrated; learned routers like RouteLLM address this with preference data and report over 2x cost reductions at matched quality. A cascade only pays off if the deferral signal is sound, so it must be validated against your own eval set, not assumed.

Watch for

• Every request hits your most powerful model, including high-volume classification or lookup tasks a small model handles.
• You have no measured deferral signal deciding when a cheap answer is good enough to keep.
• Cost scales linearly with traffic and the easy majority of queries dominates the bill.

Apply it

1. Answer first with the cheapest model that clears your eval bar, and escalate only on failed or low-signal cases.
2. Build a deferral check (a validator or learned router) rather than trusting the model's self-reported confidence.
3. Validate the cascade against a labeled eval set to confirm escalated cases are the ones that actually needed the strong model.

Sources and further reading

• FrugalGPT: Using LLMs While Reducing Cost and Improving Performance · Chen, Zaharia, Zou, 2023
• RouteLLM: Learning to Route LLMs with Preference Data · Ong et al., 2024

The principle

A multi-agent research system can burn roughly 15× the tokens of a single chat, and token usage alone can explain most of the performance variance. That means multi-agent only makes economic sense when the task's value is high and the work genuinely parallelizes. For most tightly-coupled work, the coordination overhead isn't worth it.

Why it happens

Anthropic reported that their multi-agent research system burned about 15x the tokens of an ordinary chat interaction, and found that token usage alone explained roughly 80% of the performance variance across their evaluations, which makes the cost-versus-value tradeoff explicit. That arithmetic means multi-agent only earns its keep on tasks that are both high-value and genuinely parallelizable, where independent sub-agents can fan out on separable threads without waiting on each other. For tightly-coupled, sequential work, the agents mostly idle on each other's outputs while the coordination overhead and duplicated context inflate the bill for no quality gain. The deeper risk Cognition documents is that splitting work across agents fragments context: actions carry implicit decisions, and sub-agents making conflicting decisions from partial views produce incoherent results, so the tax is paid in both tokens and reliability.

Watch for

• The work is sequential or tightly coupled, so sub-agents mostly wait on each other rather than running in parallel.
• Token cost has jumped severalfold after splitting into multiple agents with no measurable quality improvement.
• Sub-agents make conflicting decisions because each sees only a fragment of the shared context.

Apply it

1. Reserve multi-agent architectures for high-value tasks that genuinely parallelize into independent threads.
2. For tightly-coupled work, keep it a single well-prompted agent rather than paying the coordination tax.
3. If you do split, share full traces and constraints across sub-agents so they do not make conflicting decisions.

Sources and further reading

• How we built our multi-agent research system · Anthropic, 2025
• Don't Build Multi-Agents · Walden Yan (Cognition), 2025

The principle

Any system's structure ends up a copy of the communication structure of the organization that built it. Applied to AI: if three teams each own a model, you'll get three agents and a brittle seam between them — whether or not the problem wanted to be split that way. The agent boundaries you ship will trace your team boundaries unless you consciously fight it.

Why it happens

Conway's 1968 observation is that any system's structure is constrained to mirror the communication structure of the organization that designed it, because the interfaces in the software get negotiated along the same lines as the conversations between the teams. Applied to agents, if three teams each own a model, you ship three agents with a brittle seam between them whether or not the problem wanted to be partitioned that way, and those seams become where production bugs concentrate. Martin Fowler frames the practical response as the Inverse Conway Maneuver: rather than fighting the law, deliberately shape teams to match the architecture you want, so the boundaries you ship reflect the problem instead of the reporting lines. The leverage point is upstream, in team and ownership structure, not in the diagram you draw after the boundaries have already been socially negotiated.

Watch for

• Agent or service boundaries line up exactly with team ownership rather than with natural seams in the problem.
• Most production bugs cluster at the handoffs between components owned by different teams.
• A task that wanted to be one coherent flow was split because no single team owned the whole thing.

Apply it

1. Before committing boundaries, check whether each one reflects the problem's structure or just your reporting lines.
2. Where a boundary serves the org chart but not the problem, reshape team ownership to match the architecture you want.
3. Treat the seams between components as the highest-risk surface and design explicit contracts there.

Sources and further reading

• How Do Committees Invent? (Conway's Law) · Melvin Conway, 1968
• Conway's Law · Martin Fowler, 2022

The principle

Agents retry — on timeouts, rate limits, transient errors — but a failed call that never returned may have already succeeded server-side. Without an idempotency key, the retry that 'fixes' a network blip silently double-charges the card, double-sends the email, or double-books the room. Safe retries require the server to dedupe.

Why it happens

The trap is the ambiguous failure: a side-effecting call can succeed on the server while the response is lost to a timeout or dropped connection, so the client sees failure and retries an operation that already happened. Without server-side deduplication this double-charges the card or double-sends the email, and at scale these ambiguous failures are routine, not rare. The standard fix is an idempotency key: a client-generated unique value sent with the request, against which the server records the outcome of the first attempt and replays that same stored result for any retry carrying the same key, so the effect occurs exactly once. Retries also need exponential backoff with jitter, because synchronized retries against a struggling dependency amplify load and can drive the cascading failure the retry was meant to survive.

Watch for

• A side-effecting tool call is retried on timeout with no key that lets the server recognize a duplicate.
• Retries fire immediately or on a fixed interval rather than with exponential backoff and jitter.
• You have seen duplicate charges, emails, or records traced to a network blip rather than a logic bug.

Apply it

1. Generate a unique idempotency key per logical action and send it on every side-effecting call so the server can dedupe.
2. Never let the agent blindly retry a non-idempotent operation without that key.
3. Retry with exponential backoff and jitter so synchronized retries do not amplify load on a struggling dependency.

Sources and further reading

• Making retries safe with idempotent APIs · AWS Builders' Library
• Idempotent requests (Stripe API Documentation) · Stripe

The principle

A downstream model or tool that's timing out doesn't get healthier by being called more — it gets worse, while your agents pile up holding open connections and burning latency budget. A circuit breaker wraps the call so that once failures cross a threshold it trips: further calls fail fast instead of hanging, giving the dependency room to recover.

Why it happens

A dependency that is timing out does not recover by being called more; the extra load deepens its overload while callers pile up holding open connections and draining their own latency budget, which is the mechanism of a cascading failure. A circuit breaker, formalized by Nygard and Fowler, wraps the call in a small state machine: it counts failures, and once they cross a threshold it opens so further calls fail fast instead of hanging, then after a cooldown it goes half-open to let a probe request test recovery before closing again. Failing fast is the point, it converts an indefinite hang into a predictable, immediate error the agent can degrade against, and it gives the sick dependency room to recover instead of being hammered. Google's SRE guidance pairs this with shedding retries and traffic upstream once total load exceeds capacity, because uncontrolled retries are a primary driver of the cascade the breaker exists to stop.

Watch for

• When a downstream model or tool slows down, your agents respond by retrying harder and connections pile up.
• A single failing dependency drags whole-run latency toward your timeout ceiling instead of failing fast.
• There is no fast-fail path: calls to a known-sick dependency hang until they time out individually.

Apply it

1. Wrap every external model and tool dependency in a breaker that opens after a failure threshold and fails fast.
2. After a cooldown, let a single probe test recovery before resuming full traffic.
3. Shed retries and traffic upstream when load exceeds capacity so retries do not amplify the cascade.

Sources and further reading

• CircuitBreaker · Martin Fowler (after Nygard, 'Release It!')
• Site Reliability Engineering, Ch. 22: Addressing Cascading Failures · Google SRE

The principle

Automation doesn't shrink the human role — it transforms it into the hardest parts: passive monitoring plus rare, high-stakes intervention. Worse, by taking over the routine work, automation erodes the very skills and situational feel the operator needs when control is finally handed back. You design away the easy 95% and leave humans the 5% they're now least equipped to handle.

Why it happens

Bainbridge's core irony is structural: the designer hands the routine cases to the machine and leaves the operator only the situations the automation could not handle, which are by definition the hardest ones. Two reinforcing mechanisms make this worse over time. First, manual skill decays without practice, so the operator who must take over in an emergency is less competent than they were on day one. Second, situation awareness drops when you are passively monitoring rather than actively controlling, a pattern Endsley later named the automation conundrum: the more reliable and autonomous the system becomes, the less the supervising human understands its state and the harder it is for them to step back in. The leftover human role is not a smaller version of the old job; it is a different and more demanding one.

Watch for

• The human in the loop only ever sees the cases the agent already failed on, with no exposure to normal runs.
• When the agent escalates, it hands over a half-finished result with no explanation of what it tried or why it stopped.
• The people meant to supervise the agent can no longer do the task manually because the agent has done it for months.

Apply it

1. Route a sample of ordinary, successful cases to the human too, not just the exceptions, so their skill and context stay warm.
2. On every handback, attach the full reasoning trace and a plain statement of exactly what is stuck and why.
3. Design the escalation moment deliberately: make it rare, unambiguous, and accompanied by enough context to act on.

Sources and further reading

• Ironies of Automation · Lisanne Bainbridge, 1983
• From Here to Autonomy: Lessons Learned From Human-Automation Research · Endsley, 2017

The principle

Given an automated aid, operators make errors of omission (missing problems it didn't flag) and commission (following its recommendation even when their own valid evidence contradicts it). Automation becomes a heuristic shortcut that replaces vigilant checking — so the agent's recommendation doesn't just inform the human, it overrides their independent judgment.

Why it happens

Automation bias is the use of an automated aid as a heuristic shortcut that replaces effortful, independent checking of the underlying evidence. Mosier and Skitka documented it as two distinct error types: omission errors, where the operator misses an event because the automation did not flag it, and commission errors, where the operator follows the automation's recommendation even when other available information contradicts it. The driver is cognitive economy: verifying the machine is work, and deferring to it is cheap, so attention quietly migrates from the raw signals to the verdict. The effect strengthens precisely when it is most dangerous, in time-critical or high-workload moments, and high reliability makes it worse rather than better because each correct call trains the operator to stop looking. Crucially, the presence of the recommendation itself changes behavior, so showing only the conclusion all but guarantees rubber-stamping.

Watch for

• Reviewers approve the agent's recommendation at near-100% rates, far faster than it would take to actually inspect the evidence.
• The interface shows the verdict prominently but buries or omits the raw signals the verdict was based on.
• Disagreeing with the agent takes more clicks or justification than agreeing with it.

Apply it

1. Present the raw evidence next to the recommendation, never the verdict alone.
2. Make disagreement a frictionless, one-step action that needs no special justification.
3. Periodically withhold the recommendation entirely so the human has to form an independent judgment.

Sources and further reading

• Does automation bias decision-making? · Skitka, Mosier, Burdick, 1999
• Automation Bias in Intelligent Time Critical Decision Support Systems · Cummings, 2004

The principle

Autonomy is a spectrum — from 'the computer suggests' to 'the computer acts then tells you' to 'the computer acts and decides whether to tell you at all'. The highest levels are unwise for consequential actions because no aid is perfectly reliable and the cost of a confident error is unbounded. Autonomy isn't one switch; it's a dial you set per action by how reversible and costly that action is.

Why it happens

Sheridan and Verplank framed automation as a graded scale rather than an on/off switch, and Parasuraman, Sheridan, and Wickens later refined this into a framework where automation can be applied at different levels across four separable stages: acquiring information, analyzing it, deciding an action, and executing it. The key insight for consequential actions is that the right level is bounded by reliability and cost: because no aid is perfectly reliable, the appropriate level of automation falls as the cost of an error rises. A high autonomy level applied uniformly means the system executes irreversible actions at the same trust setting it uses for trivial ones, so a single confident error carries unbounded downside. The correct design is per-action, set by reversibility and blast radius, not a single global switch, which also sidesteps the opposite failure of forcing human confirmation on cheap reversible actions and breeding confirmation fatigue.

Watch for

• The agent uses one autonomy setting for everything, so resending a receipt and issuing a large refund run through the same path.
• Irreversible or high-cost actions execute before any human can see them.
• Humans are buried in approval prompts for trivial, reversible actions, training them to click through blindly.

Apply it

1. Classify each action by reversibility and blast radius before deciding its autonomy level.
2. Let cheap, reversible actions run fully autonomously and gate costly or irreversible ones to propose-and-confirm.
3. Tune the dial per action rather than flipping one global approval flag for the whole agent.

Sources and further reading

• Human and Computer Control of Undersea Teleoperators · Sheridan & Verplank, 1978
• A Model for Types and Levels of Human Interaction with Automation · Parasuraman, Sheridan & Wickens, 2000

The principle

Flexible, multi-mode automation produces 'automation surprises' — the system does something unexpected because the operator lost track of which mode it was in, what it would do next, and why. As autonomy grows, the human's job shifts to tracking its state, and every hidden mode transition becomes a latent failure path. An agent that silently changes how it behaves leaves its supervisor one step from being wrong about it.

Why it happens

Sarter and Woods traced automation surprises to a specific breakdown: a loss of mode awareness, where the operator no longer tracks the system's current and future status and behavior. The mechanism is a mismatch between the operator's mental model of what the automation will do and what it actually does, combined with an interface that fails to make state transitions salient. Their study of airline crews found these surprises cluster in non-normal and time-critical situations and often follow uncommanded or silent mode changes that the automation never announced. For an agent, every hidden shift, from planning to acting, from one tool policy to another, is a latent mode transition that leaves the supervisor reasoning about a system that no longer exists. The supervisor is then one step away from being wrong about everything the agent does next, not because the agent misbehaved but because its state became invisible.

Watch for

• The agent changes behavior, such as switching from drafting to executing, without surfacing that the switch happened.
• A supervisor cannot answer what mode is it in and what will it do next from the current display.
• Post-incident reviews repeatedly conclude I thought it was still just proposing.

Apply it

1. Keep the current mode, active constraints, and next intended action continuously visible.
2. Make every mode transition an explicit, loud event the supervisor must see, never a silent switch.
3. Treat any uncommanded change in behavior as a defect to surface, not an optimization to hide.

Sources and further reading

• How in the World Did We Ever Get into That Mode? · Sarter & Woods, 1995
• Team Play with a Powerful and Independent Agent: Automation Surprises on the A-320 · Sarter & Woods, 1997

The principle

Each agent can be flawless in isolation and the system still breaks — because the bug lives between them: what got passed, what got dropped, who owned the state. Sub-agents don't inherit context automatically; anything not explicitly handed over simply doesn't exist on the other side.

Why it happens

In a multi-agent system the failure surface shifts from inside each agent to the boundaries between them, because nothing crosses a boundary unless it is explicitly passed. The empirical study behind the MAST taxonomy analyzed over 200 traces across seven multi-agent frameworks and found that a large share of failures were not single-agent reasoning errors but coordination breakdowns: inter-agent misalignment, dropped or misunderstood information, and missing verification of what was handed over. This mirrors the human-factors finding that handoffs fail when common ground, the shared context both sides assume, is not actually established. A sub-agent has no access to the constraints, sources, or state the orchestrator never serialized into the message, so a constraint like EU market only simply does not exist on the far side. The bug is invisible because each agent, judged alone, did exactly what it was told.

Watch for

• A downstream agent produces output that violates a constraint the upstream agent clearly knew about.
• Nobody can say which agent owns a given piece of state, so it gets dropped or duplicated.
• What crosses a boundary is assumed correct and never validated on the receiving side.

Apply it

1. Define an explicit contract at every boundary listing exactly what must be passed.
2. Hand over the full constraint set and source set deliberately rather than assuming context is inherited.
3. Validate incoming data on the receiving side instead of trusting it survived the trip.

Sources and further reading

• How we built our multi-agent research system · Anthropic, 2025
• Why Do Multi-Agent LLM Systems Fail? · Cemri et al., 2025

The principle

People extend an agent freedom the way they extend it to a new hire — incrementally, on reversible things first, widening the leash only as it proves itself. Both failure modes are real: over-trust causes misuse, under-trust causes a good capability to be abandoned. Reliance tracks the perceived reliability the system reveals, not just its true reliability.

Why it happens

Lee and See model reliance as a function of trust, and the central design goal is calibration: making perceived trustworthiness match actual reliability, so people rely on the system where it is strong and not where it is weak. Miscalibration runs both ways and both are costly. Over-trust produces misuse, where people lean on the system past its competence, while under-trust produces disuse, where a genuinely capable aid is abandoned. The under-trust failure has a sharp empirical edge: Dietvorst and colleagues showed algorithm aversion, where people lose confidence in an algorithm faster than in a human after seeing it make the very same error, and will then choose an inferior human forecaster instead. Because reliance tracks the reliability the system reveals rather than its true reliability, calibration is achieved by exposing where the agent is confident and where it is guessing, not by hiding its limits.

Watch for

• The agent is given broad write access to high-stakes systems before it has a track record on reversible ones.
• Every single action is funneled through manual approval, and the team is quietly abandoning the tool from fatigue.
• The agent presents strong and shaky outputs with identical confidence, giving users no basis to calibrate.

Apply it

1. Start the agent on low-stakes, reversible actions and widen its blast radius only as reliability is proven.
2. Surface where the agent is reliable versus where it is guessing so users rely on it exactly that far.
3. Avoid both extremes: neither hand it production write access on day one nor gate every trivial action behind approval.

Sources and further reading

• Trust in Automation: Designing for Appropriate Reliance · Lee & See, 2004
• Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err · Dietvorst, Simmons & Massey, 2015

The principle

An agent with no legitimate way to say 'I'm stuck' or 'hand this to a human' will invent a path instead. Cornered without an exit — or forced to fill a required field it has no answer for — it fabricates something plausible rather than admitting the gap. A confident hallucination is the default when honesty isn't an option.

Why it happens

A model is a fluent continuation engine, not a truth-teller, so when honesty is not an available output it produces the most plausible-looking token sequence instead, which is a confident fabrication. The survey literature on hallucination frames this as the model generating content that is fluent and confident but unsupported by or in conflict with the available evidence, and a major contributing factor is the pressure to always produce an answer. If a required field must be filled or a workflow offers no I am stuck branch, the only path forward the model has is to invent something that satisfies the schema. The fix is to make abstention a first-class, low-cost option: a nullable field, an explicit unknown value, or an escalate-to-human action. When I do not know is a valid and easy answer, the model no longer has to choose fabrication to keep moving, and you trade confident errors for honest, actionable gaps.

Watch for

• Required fields are never empty, even on inputs where the answer genuinely cannot be known.
• The agent has no action that means hand this to a human or I cannot do this.
• Plausible but wrong values appear in exactly the cases where the source data was missing or ambiguous.

Apply it

1. Give the agent a first-class way out: a nullable field, an explicit unknown, or an escalate-to-human action.
2. Make abstaining cheap and explicitly encouraged rather than something the agent must avoid.
3. Treat a confident answer on missing data as a failure mode to detect, not a success.

Sources and further reading

• Reduce hallucinations (Anthropic Docs) · Anthropic
• Survey of Hallucination in Natural Language Generation · Ji et al., 2023

The principle

Without an external signal, a model largely fails to self-correct its own reasoning — and often makes correct answers worse by second-guessing them. The model that produced a flawed plan is the same one judging it, with the same blind spots. Real correction needs an outside signal: a tool result, a test that runs, a different model. 'Reflect and try again' on the same model with no new information is theater.

Why it happens

Self-correction fails because the model judging an answer is the same model that produced it, carrying the identical blind spots, so it has no new information to correct against. Huang and colleagues found that without an external signal, models largely cannot self-correct their reasoning and often degrade correct answers by second-guessing them. Stechly, Valmeekam, and Kambhampati pushed further on reasoning and planning tasks, showing that the model's own self-verification is unreliable and that gains attributed to reflection largely vanish or come from an external verifier, not introspection. The shared lesson is that reflect and try again on a fixed model with no fresh input is theater: the second pass samples from the same flawed distribution. Genuine correction requires an outside signal, such as a tool result, a test that actually runs, or a separate model with no memory of the original attempt.

Watch for

• Your correction step is just review your work and fix any bugs with no new input introduced.
• The agent confidently rewrites a correct answer into a wrong one after being asked to reflect.
• A corrected output is trusted without any external check ever having run.

Apply it

1. Separate generation from judgment: never let the producing instance be the sole grader.
2. Feed an external signal into the correction loop, such as a test that runs, a tool result, or compiler output.
3. When using a model to judge, give it a fresh instance with no memory of the original reasoning.

Sources and further reading

• Large Language Models Cannot Self-Correct Reasoning Yet · Huang et al., 2023
• On the Self-Verification Limitations of LLMs on Reasoning and Planning Tasks · Stechly, Valmeekam & Kambhampati, 2024

The principle

When findings get summarized and re-summarized, the claim survives but its source, its date, and its uncertainty quietly fall away — until you're holding an assertion you can't verify or defend. Two sources disagreeing isn't noise to flatten; it's signal to keep. A fact without provenance is a rumor with good posture.

Why it happens

Each summarization pass is a lossy compression that preserves the surface claim while discarding the metadata that lets you trust it: the source, the date, the uncertainty, and any disagreement between inputs. After a few hops a hedged, dated, conflicted finding collapses into a flat assertion with the same confident shape as a verified fact, which is why a claim without provenance is a rumor with good posture. Research on grounded generation treats this as a measurable property, attribution, where every claim should be traceable back to a supporting source, and benchmarks like ALCE evaluate generated text on exactly that citation faithfulness. The practical fix is to carry the full tuple, claim plus source plus date plus confidence, through every transformation rather than only the claim. Conflicts in particular must be preserved with both sides attributed, because a disagreement between two sources is signal about reliability, not noise to be flattened into a single winner.

Watch for

• A final report states a figure or claim with no source, date, or confidence attached.
• Two sources that disagreed upstream have silently become a single confident number downstream.
• You cannot trace a claim in the output back to the specific document it came from.

Apply it

1. Carry claim, source, date, and confidence together through every summarization and transformation step.
2. When sources conflict, keep both values with their attributions instead of silently picking a winner.
3. Require that every claim in the final output be traceable back to a specific supporting source.

Sources and further reading

• Effective Context Engineering for AI Agents · Anthropic
• Enabling Large Language Models to Generate Text with Citations (ALCE) · Gao et al., 2023