Law 13 · Retrieval & Memory
Relevant Beats Plenty
Near-misses poison context worse than random noise.
The principle
Counterintuitively, documents that are topically related but don't answer the question are more harmful than clearly irrelevant ones — they look plausible and pull the generator toward wrong-but-adjacent answers. Stuffing more 'kind of relevant' chunks into context degrades accuracy rather than improving coverage. Precision at the top beats breadth.
Why it happens
A distractor that shares vocabulary and topic with the query but lacks the answer is dangerous because it scores high on the same surface features the generator uses to decide what is relevant, so the model treats it as evidence and anchors a plausible but wrong answer to it. Clearly off-topic noise is comparatively safe because the model can recognize and discard it, which is why near-misses degrade accuracy more than random noise of the same volume. Controlled experiments on retrieval for RAG found this counterintuitive result directly: adding related-but-irrelevant passages hurt answer accuracy while injecting unrelated random documents could leave it stable or even help, meaning precision at the top of the ranking matters more than raw coverage. Padding context with more kind-of-relevant chunks therefore trades a small recall gain for a larger precision loss.
Watch for
- Raising top-k to improve coverage makes answers worse, not better.
- Wrong answers are adjacent to the truth, like the right product family but the wrong model number.
- Context is filled with many topically similar chunks and no reranking step trims them.
In practice
To improve coverage you bump top-k from 5 to 20, and accuracy drops, because the 15 new chunks are all topically adjacent: same product line, wrong model number, and they pull the answer toward a plausible lie. Clearly irrelevant chunks get ignored, but near-misses get believed. Do not pad context for recall's sake. Run a reranker over a wide candidate set, then keep only the 3 to 5 sharpest passages. A tight context beats a stuffed one.
Apply it
- Retrieve a wide candidate set but rerank and keep only the few highest-precision passages.
- Tune for precision at the top of the ranking rather than maximizing recall at any cost.
- Drop topically similar chunks that do not directly answer the query instead of including them for safety.
The takeaway
Optimize for precision, not recall-at-any-cost. Aggressively rerank and filter out distractor chunks — a smaller, sharper context beats a padded one.