The Unit Test Passed and the Agent Was Still Wrong
A regional payroll processor I work with runs an AI agent that handles overtime classification across multiple state jurisdictions. The agent was updated to reflect a new interpretation of a blended-rate calculation. Every unit test passed. The logic was sound in isolation. The agent went to staging and looked clean.
When we ran the replay suite against six months of real historical payroll runs, the agent misclassified overtime for a specific employee category that only appeared in three runs out of several hundred. Those employees worked split shifts across two cost centers, and the new blended-rate logic broke a downstream rounding step that the old logic had handled implicitly. The unit tests never saw that case because nobody thought to write one for it. The historical runs had it, because real work is messier than test cases.
We caught it before it touched a single paycheck. That is the whole argument for replay testing in one paragraph.
Why Quiet Rule Changes Are the Hardest Failures to Catch
Loud failures are easy. The agent throws an error, the pipeline stops, someone gets paged. Quiet failures are the ones that ship. A quiet rule change is when the agent still produces a confident, well-formatted output that is numerically wrong in a way that only becomes visible when you compare it against what the same input produced three months ago.
In finance operations this happens constantly. Tax tables update. Jurisdiction thresholds shift. A benefits vendor changes how they report hours. None of these changes break the agent in any obvious way. The agent reasons through the new inputs using the same logic it always used, arrives at a number, and moves on. The number is wrong. The agent does not know it is wrong. No alarm fires.
In manufacturing I have seen the same pattern with shift differential calculations and with material cost allocations that depend on commodity price tiers. The tier boundaries change, the agent applies the old mental model to the new boundaries, and the output drifts. The drift is small enough per transaction that it hides in noise until someone runs a monthly reconciliation and finds a gap they cannot explain.
What a Replay Suite Actually Does
A replay suite is not complicated in concept. You store the inputs and the outputs from real past runs. When you update the agent, you feed it the same historical inputs and compare the new outputs to the old ones. Any divergence is a candidate for review.
The hard part is not the comparison. The hard part is deciding what counts as an acceptable divergence and what counts as a regression. If you tightened a rounding rule on purpose, you expect some outputs to change. If you fixed a known bug, you expect the outputs for affected cases to change. Everything else that changes is a signal worth investigating.
The practical setup I use in production looks roughly like this.
- Capture real runs in a structured log that includes the full input context, the agent reasoning trace, and the final output.
- Tag each run with the version of the agent that produced it.
- On every agent update, run the full historical set through the new version in a sandboxed environment.
- Diff the outputs and flag divergences above a configurable threshold.
- Route flagged divergences to a human reviewer with the old output, the new output, and the reasoning trace for both.
- Require sign-off on any divergence before the new version promotes to production.
This is not a novel idea. It is what software engineers do with snapshot testing for UI components. The difference is that AI agent outputs have more surface area and more ways to be subtly wrong, so the review step matters more.
The Distribution Sector Case That Made Me Take This Seriously
Before I formalized this approach I learned the lesson the expensive way in a distribution operation. The agent handled freight cost allocation across a network of regional carriers. We updated the agent to handle a new carrier contract structure. Tests passed. The agent went live.
Three weeks later the finance team found that a subset of shipments involving a specific lane combination had been allocated to the wrong cost center. The amounts were not large per shipment but they had accumulated. The root cause was that the new contract logic changed how the agent interpreted a field that appeared in the old carrier records too, and the old records were still flowing through for shipments booked before the contract change.
A replay suite would have surfaced this on day one. The historical runs included plenty of the old-format records. The new agent would have processed them differently. The diff would have flagged it. Instead we found it in a reconciliation meeting.
After that I made replay testing a hard requirement for every agent update in every production system I operate. Not a recommendation. A requirement.
What Replay Testing Does Not Solve
I want to be honest about the limits because this approach gets oversold once people see it work.
Replay testing catches regressions on cases you have already seen. It does not catch failures on genuinely novel inputs. If a new jurisdiction appears in your payroll data that has never appeared before, your historical suite has nothing to say about it. You still need unit tests for known edge cases, and you still need human review for novel situations.
Replay testing also does not validate that the original outputs were correct. If the agent was wrong six months ago and you are comparing new outputs to those old wrong outputs, a regression suite will flag the correct new output as a divergence. This is why I always pair the replay suite with a periodic ground-truth audit where a domain expert reviews a sample of historical outputs and marks them as correct or incorrect. The replay suite tells you something changed. The ground-truth audit tells you which direction is better.
And replay testing does not remove the need for monitoring in production. Models can behave differently on live data than on historical data for reasons that are hard to predict. The replay suite is a gate before deployment. Production monitoring is what catches the things that slip through.
Field Operations and the Case for Sector Specific Suites
In field operations I work with agents that handle work order routing, technician scheduling, and parts procurement. The replay suite pattern applies here too but the divergence criteria are different. A payroll output is a number. A work order routing output is a sequence of decisions with dependencies, and a change in one step can cascade through the rest.
For these systems I build sector-specific replay suites that capture the full decision chain, not just the final output. When the agent changes, I compare the reasoning trace step by step, not just the endpoint. This catches cases where the agent arrives at the same final answer by a different path that would break under conditions not present in the historical data.
A field operations client in utilities had an agent update that changed how the agent prioritized emergency work orders when technician availability was constrained. The final routing outputs for most historical runs looked identical. But the reasoning traces showed the agent was now deprioritizing a specific class of infrastructure inspection that had never been in conflict with emergency work during the historical period. We flagged it, reviewed it with the operations team, and they confirmed it would have caused compliance problems during high-demand periods. The historical data did not contain a high-demand period. The trace review did.
The Practical Takeaway
If you are running AI agents in production and your test strategy is unit tests plus staging review, you are flying with one instrument. Unit tests check the logic you thought to test. Staging review checks the cases you thought to create. Neither of them checks the full texture of real operational history.
Start by logging every production run with enough detail to replay it. That is the foundation. Once you have six to eight weeks of real runs, build the comparison infrastructure and run it against your next agent update before it promotes. You will almost certainly find something you did not expect. That finding, before it ships, is the return on the investment.
The payroll case I opened with was not a dramatic near miss. It was a quiet one. A few hundred employees, a specific shift pattern, a rounding step that nobody remembered was load-bearing. The replay suite found it in forty minutes. A reconciliation audit would have found it six weeks later. The difference between those two timelines is the difference between a fix and an apology.