Regression Testing
Our reliability layer protects known working behavior when models, prompts, retrieval, tools, or workflows change.
Detecting the trade
Any of those components can improve one part of the system while degrading another. Regression testing exists to detect that trade before the change becomes production behavior.
Regression surface
Changes can include the model, the prompt, retrieval, the workflow, and the tools. A regression suite preserves representative cases that the system is already expected to handle correctly.
- Current production behavior
- Known-good regression set
- Candidate change
- Run the same regression set
- Compare behavior
- Pass, investigate, or block
The tests can include exact outputs where exactness is required and rubric or evaluator-based checks where the workload is inherently interpretive.
Preserve difficult wins
Regression sets are particularly valuable for failures that required effort to fix. Once an edge case, tool interaction, retrieval failure, or structured-output problem is resolved, it can become a permanent test case rather than institutional memory held only by the engineers who debugged it.
This turns production learning into reusable reliability infrastructure.
Not only models
A model upgrade is an obvious regression risk, but the surrounding stack matters equally. Changing chunking can alter retrieval. Modifying a prompt can change tool selection. Updating an API can change workflow state. Tightening permissions can block an execution path that previously succeeded.
The regression suite should therefore test the relevant system behavior, not only raw model output.
Limits
Regression testing protects what has been encoded into the test set. It does not detect every new failure mode. We combine regression tests with fresh evaluations, failure detection, adversarial review, and production observability.
A new capability should not silently erase a capability that already worked.
Regression ownership
A production regression suite should grow with the system. When an important failure is discovered and corrected, the corresponding case can be added to the suite so the same defect cannot silently reappear after a future model, prompt, retrieval, or workflow change. That turns debugging history into durable infrastructure.
Regression cases can also be tagged by component or consequence. If a retrieval change fails only tool-use tests, or a model upgrade fails only a high-consequence subset, engineers can see the shape of the regression rather than only an aggregate pass rate.
The suite should remain representative rather than infinitely additive, but important historical failures deserve permanent protection when their recurrence would matter operationally.