When the Commonwealth Bank of Australia announced in July 2025 that an AI voice bot would replace 45 customer-service roles, the staff union reported rising call volumes. [1] By August, the bank had reversed its decision, acknowledging the roles were still needed. The episode captures the central problem that RehearsalBench was built to measure: organizations now ask AI systems what will happen if they change how they operate, and the systems answer with confidence, but the answer is often wrong in ways that matter.
The benchmark, developed by researchers at Afterflow Inc. and an independent contributor, places large language models in simulated bank contact centers built from real call-level records. Ten models made 280 attempts across 14 organizations. The best model, GPT-6 Astra, improved on doing nothing by 6.8% of the no-change cost and made 77% less forecast error than a forecaster who always predicts no effect. [1] Those numbers sound like progress. They are also the setup for a finding that undercuts the entire premise of asking AI to make operational judgments.
The Direction Is Right, the Magnitude Is Wrong
Every model, from every developer, forecast the direction of every specified effect correctly. Deploy a voice agent, and repeat contacts go up. Cut staff, and waiting times rise. The models understood the sign of the change. What they could not do was size it.
Models overstated a voice agent’s effect on repeat contacts approximately five-fold. The simulated effect, drawn from documented ranges, was small. The models predicted a flood. This is not a small calibration error. A five-fold overestimate of repeat contacts turns a marginal intervention into an apparent disaster, or a modest improvement into a transformation.
The pattern held across models and across effects. The errors were not random. They leaned in the same direction, systematically exaggerating the consequences of the same interventions. A benchmark that only checked whether models got the sign right would have declared victory. RehearsalBench checked the magnitude, and the magnitude is where judgment lives.
The Refusal That Never Comes
Two of the 14 test banks were constructed so that no available plan reliably outperforms current operations. They test whether models recognize when the best decision is no decision. Every model failed.
This is the finding that directly addresses the question of what AI makes superfluous. A human manager who has run a contact center for years knows when to leave it alone. That knowledge is not a formula. It is a judgment built from experience with failed initiatives, from a sense of organizational fatigue, from the memory of the last time a vendor promised the world and delivered a reshuffled roster. The models have no such memory. They were given a problem and asked to solve it, and they solved it, even when the problem was that there was nothing to solve.
The absence of restraint is not a bug in the models’ reasoning. It is a feature of how they are asked to reason. The task in phase one of the benchmark is to choose a plan. Keeping current operations is one option among many, but the framing invites intervention. A model that returns “do nothing” looks like a model that failed to find the answer. A model that proposes a staff cut, a voice agent, and a coaching program looks like a model that did its job. The incentive structure of the evaluation, and of the real-world requests that inspired it, rewards action over restraint.
What the Benchmark Actually Measures
RehearsalBench separates two tasks that are usually conflated. In phase one, a model reads management reports, a roster, a cost sheet, and stakeholder statements. It may purchase evidence, such as a short pilot, at a cost charged to its plan. It then ranks bottlenecks, chooses a plan, and forecasts the plan’s outcomes. In phase two, a fresh conversation gives every model the same list of assigned interventions, and each model forecasts the effect of each intervention on five outcomes: abandonment rate, mean wait, average handling time, repeat-contact rate, and staff cost.
The separation matters because it allows forecasts to be compared even when models recommend different plans. If model A proposes a voice agent and model B proposes a staffing change, their decision values are not directly comparable, because they are solving different problems. But their forecasts of the same assigned interventions are comparable, because they are predicting the same counterfactual. This design choice is what makes the benchmark’s central finding possible: the models agree on direction and disagree on size, and the disagreement is systematic.
Decisions are scored in dollars at the stakeholders’ stated valuations. A plan that reduces repeat contacts is worth more if the bank’s finance team assigns a high cost to those contacts. A plan that reduces staff cost is worth more if the bank is under a hiring freeze. The benchmark defines a utility function with bank-specific stakeholder valuations rather than a single universal objective. It asks each model to read the stakeholder statements and infer what the organization values. This is another place where human judgment is supposed to matter, and another place where the models’ performance reveals the limits of their understanding.
The Real-World Test
To check whether the benchmark’s rankings mean anything outside the simulator, the researchers had twelve models forecast 33 real organizational changes with known outcomes. The changes were disguised, and the models were checked for recognition, to guard against the possibility that they had memorized the outcomes from training data. Under verdict rules fixed in advance, the benchmark’s forecast ranking agreed with the real-world ranking, with a Spearman correlation of 0.92 and a 95% lower bound of 0.50. [1] The researchers are careful about what this correlation means. They report it alongside an inconclusive first test and disclose that some analysis choices were made after outcomes were seen. They also present evidence that part of the agreement reflects how well models size their uncertainty, not just how accurately they predict point estimates. A model that gives wide prediction intervals may score better on an interval-based metric even if its central estimate is off, because the wide interval is more likely to contain the true value. The correlation, in other words, may be measuring a kind of calibrated humility rather than predictive accuracy.
That distinction matters for the question of what AI makes superfluous. If the models’ advantage lies in knowing how uncertain they should be, then, in the authors’ framing, the skill being automated is not judgment but hedging. A human expert who says “this will probably help, but I’m not sure how much” is doing something the models can also do. A human expert who says “don’t do this, I’ve seen it fail before” is doing something the models cannot do, because they have no before.
What the Models Cannot See
The simulated banks are built from real call records, but the models never see the records. They see management reports, a roster, a cost sheet, and stakeholder statements. They see what a manager would see, not what a researcher would see. This is a deliberate design choice, and it is the source of the benchmark’s most interesting failures.
A manager who has worked in a contact center knows things that are not in the reports. The models have no access to this tacit knowledge. They have only the documents, and the documents are incomplete.
The benchmark does not penalize models for lacking tacit knowledge, because the benchmark cannot measure what it does not provide. But the absence of tacit knowledge is precisely what makes the models’ confident recommendations dangerous. A model that recommends a staff cut based on the reports does not know that the reports omit the fact that the staff are already stretched. The model sees a problem and solves it, and the solution is only as good as the problem definition.
The Intervention That Was Not Recommended
The benchmark’s assigned interventions include a cut of about two staff members, an AI voice agent, callbacks offered when the expected wait exceeds five minutes, AI assistance for all staff, self-service content for the largest simple call type, two hours of coaching per staff member per month, and, in banks whose staff form a single team, two staff members reserved for specialist call types. These are the levers a manager can pull. The models were asked to forecast what happens when each lever is pulled.
The models got the direction right. They got the size wrong. They never recognized, in any case, the value of doing nothing. The pattern is consistent with a system that has learned to associate interventions with outcomes but has not learned to calibrate the strength of those associations. It knows that voice agents cause repeat contacts. It does not know how many.
This is the gap that human judgment fills. A manager who has deployed a voice agent before knows that the repeat-contact rate goes up, but not by five times. They know from experience that the effect is real but modest. They know that the cost of the voice agent is worth bearing if the containment rate is high enough, and they know how to read the vendor’s claims with skepticism. The model has no such experience. It has training data, and training data is not the same as having been there.
The Benchmark as a Rehearsal
The researchers describe RehearsalBench as a way to rehearse AI-recommended operational changes before they reach real customers and staff. The metaphor is apt. A rehearsal is not a performance. It is a safe space to make mistakes. The benchmark allows models to make mistakes in a simulated bank, where the cost is measured in benchmark points rather than in lost customers and demoralized staff.
But a rehearsal is only useful if the performers learn from it. The benchmark does not train the models. It evaluates them. The researchers note that the same environment could be used to train agents with reinforcement learning in future work, but that is not what the benchmark does now. It measures the models’ current capabilities, and the measurement reveals a consistent pattern of overconfidence in the size of effects and an inability to recognize when no intervention is warranted.
The question of what AI makes superfluous is not answered by the benchmark’s headline numbers. The 6.8% improvement is real, but it is small. The models’ forecasts were better than a no-effect baseline, but they were wrong in systematic ways. The benchmark does not show that AI can replace a manager’s judgment. It shows that AI can approximate the direction of that judgment while failing at its magnitude and its restraint.
The Part That Cannot Be Automated
The part that cannot be automated is the part that says “don’t.” The models never looked at a stable operation and said, “This is fine.” They never weighed the cost of disruption against the benefit of improvement and decided that the disruption was not worth it. They proposed plans, because that is what they were asked to do.
A human manager who has run a contact center for a decade has a different relationship with the question. They have seen initiatives fail. They have seen staff leave because of poorly planned changes. They have seen customers complain because a new system was rolled out before it was ready. They have a visceral sense of the cost of change, and that sense informs their judgment about when to change and when to leave things alone.
The models have no such sense. They have data, and data is not the same as experience. The benchmark measures the gap between the two, and the gap may be widest where it matters most: in the decision to do nothing.
What the Correlation Does Not Show
The real-world validation is the benchmark’s most ambitious claim. Twelve models forecast 33 real organizational changes, and the benchmark’s ranking agreed with the real-world ranking. The correlation is high, and the lower bound is above zero. The researchers are careful to note that the test was pre-registered and that recognition checks were used to guard against memorization. They are also careful to note that part of the agreement reflects how well models size their uncertainty.
This caveat is important. A model that gives wide prediction intervals will score well on an interval-based metric even if its point estimates are off. The correlation between the benchmark ranking and the real-world ranking may be measuring the models’ ability to hedge, not their ability to predict. A model that says “the effect will be somewhere between negative ten and positive ten” is more likely to be right than a model that says “the effect will be exactly five,” even if the true effect is zero. The wide interval is a form of insurance, and insurance is not the same as insight.
The researchers do not claim that the correlation proves the benchmark’s validity. They present it as evidence, alongside the caveats. The evidence is suggestive, but it is not conclusive. The real-world test is a step toward validating the benchmark, not a final verdict.
The Cost of Getting It Wrong
The Commonwealth Bank episode is a reminder of what is at stake. The bank announced that an AI voice bot would replace 45 roles. The staff union reported rising call volumes. The bank reversed its decision. The reversal cost money, time, and credibility. A benchmark that could have predicted the outcome would have been valuable. A benchmark that predicted the wrong direction would have been worse than useless.
RehearsalBench does not predict the outcome of a specific change at a specific bank. It measures whether models can predict the effects of interventions in simulated banks. The models get the direction right and the size wrong. They never recommend doing nothing. They overstate the effect of voice agents on repeat contacts by a factor of five. These are the findings, and they are not reassuring.
The benchmark’s value lies in what it reveals about the limits of AI judgment. The models can read reports, identify bottlenecks, and propose plans. They cannot calibrate the size of effects, recognize when no intervention is warranted, or weigh the cost of disruption against the benefit of improvement. These are the skills that human managers bring, and they are the skills that the benchmark shows are not yet automated.
The Judgment That Remains Human
The question of what AI makes superfluous is answered, in part, by what it cannot do. The models cannot say “don’t.” They cannot look at a stable operation and recognize that stability is an achievement. They cannot weigh the cost of change against the benefit of improvement and decide that the cost is too high. They cannot bring the memory of past failures to bear on a present decision.
These are not technical limitations that will be solved by more data or better training. They are limitations of the task itself. A model that is asked to choose a plan will choose a plan. A model that is asked to forecast an effect will forecast an effect. The framing of the question determines the shape of the answer, and the framing does not include the option of refusing to answer.
A human manager can refuse. They can say, “This is not the right time.” They can say, “We tried this before, and it did not work.” They can say, “The reports do not tell the whole story.” These refusals are not failures of reasoning. They are exercises of judgment, and judgment is what the benchmark shows the models lack.
The Number That Reverses Everything
The 6.8% improvement is the headline number, and it sounds like progress. But the same model overstated a voice agent’s effect on repeat contacts by a factor of five. It never recommended holding off when doing nothing was the right choice. It made forecasts that were directionally correct and quantitatively wrong.
The 6.8% improvement is real, but it is measured against a baseline of doing nothing. The model that achieved it did so by choosing a plan that beat the baseline in the simulated bank. The model did not know that the plan would work. It did not know that the effect would be the size it predicted. It did not know that the plan was better than no plan. It chose a plan because that is what it was asked to do, and the plan happened to beat the baseline.
The benchmark’s most important finding is not the 6.8% improvement. It is the five-fold overestimate, the absence of restraint, and the gap between knowing the direction of an effect and knowing its size. The models can propose. They cannot judge. The judgment remains human, and the benchmark shows why.
