Regress or Repeat

The principle

LLM systems are non-deterministic and globally coupled — a prompt tweak to fix one case silently breaks three others. Rerunning real production examples against a new prompt is the only way to know you didn't break what already worked. Without a regression suite you're trapped in a whack-a-mole loop, re-discovering the same failures release after release.

Why it happens

LLM systems are non-deterministic and globally coupled: the same prompt can yield different outputs across runs even at temperature zero because batching and floating-point execution on parallel hardware are not bit-reproducible, and a study across five models and eight tasks found accuracy varying by up to 15% across nominally identical runs. On top of that run-to-run variance, the prompt is a single shared control surface, so a change that fixes one case routinely shifts behavior on unrelated cases that share the same instructions. Together these mean you cannot reason your way to confidence that a fix is safe; you have to re-run the real prior cases and observe. Without a regression suite that captures every fixed bug as a permanent case, you are stuck in a whack-a-mole loop, re-discovering the same failures release after release.

Watch for

• A bug you fixed last release has reappeared because nobody re-ran the old case.
• A prompt tweak aimed at one case silently broke a different, unrelated case.
• You ship prompt or model changes without re-running the previously passing examples.

Apply it

1. Turn every fixed bug into a permanent regression case with its expected output.
2. Run the full regression suite on every prompt and model change before shipping.
3. Because outputs vary run to run, evaluate over repeated runs rather than trusting a single pass.

Sources and further reading

• What We Learned from a Year of Building with LLMs · Yan, Husain, et al., 2024
• Non-Determinism of Deterministic LLM Settings · Atil et al., 2024