Across 4,500 task executions, swapping in a faster decision model cut the cost and latency of every single decision an AI agent made — and still, in one agent, the average cost of actually finishing a job went up. [1] That is the finding at the center of a new empirical study on what happens when you make an agent’s thinking cheaper without changing anything else about how it works.
The study, available as a preprint (arXiv — Paper), tests five different agent families in two environments: household-style embodied tasks and mobile app navigation. [1] Each agent ran in its original configuration and then again with Jev, a decision model built by TypeSafe AI, handling the bounded choices — which action to take next from a supplied list. The setup is deliberately narrow. Jev does not reason about the task or plan. It picks from candidates the surrounding workflow hands it.
That narrowness appears to be the point. If you can make the repetitive selection step faster and cheaper, the logic goes, the whole agent should get faster and cheaper. The study tests whether that logic holds.
The savings are real and they are local
Every one of the five agents saw decision cost and latency drop substantially when Jev took over action selection. This is not ambiguous. The model is fast, it is inexpensive, and it does exactly what its interface promises: map a state and a question to a typed choice with probabilities over the supplied candidates.
The authors ran each configuration three times on the same tasks, producing the 4,500 executions that anchor the analysis. They measured task effectiveness, decision-level resource use, task-level resource use, and execution behavior — jointly, not in isolation. That joint measurement is what separates this study from a simple benchmark comparison. It is not enough to know that each decision costs less. The question is whether the task costs less.
On the embodied tasks, task success stayed close to baseline levels. The agents still completed what they were asked to complete, roughly as often as before. On mobile app tasks, success declined more substantially in two of the agents. The decision model appears not to have been making worse choices in some abstract sense. It was making choices that, in sequence, led to worse outcomes.
Where the savings go
Auxiliary reasoning and additional interactions absorbed the decision savings. This is the mechanism the study identifies, and it is worth sitting with. When the decision model picks an action, the surrounding workflow still has to reason about what that action means, still has to interact with the environment to execute it, still has to interpret the result. Those steps do not get cheaper just because the selection step did. In some workflows, they got longer.
Failed recovery and premature completion limited successful delivery. An agent that picks a wrong action has to notice the error, backtrack, and try again. That recovery process costs resources. If the decision model is less reliable at picking the right action in context — even if it is faster — the recovery costs can exceed the per-decision savings. Premature completion is the mirror image: the agent decides it is done before it actually is, and the task ends without the requested outcome.
The study found that mean cost per successful outcome increased in one agent even as spending per attempt fell. [1] In T3A, cost per delivered success rose 7.7%, from 68.70 to 73.97 mUSD, while cost per attempt fell 17.1%. This is the finding that matters most. It is not a paradox. It is arithmetic, in effect. If success rates drop enough, the cost of the failures — the attempts that consumed resources and delivered nothing — gets distributed across the successes that remain. A cheaper attempt that fails more often can be more expensive per delivered result than a costlier attempt that succeeds more reliably.
What the traces show
The authors systematically annotated execution traces and analyzed recovery outcomes. This is where the study moves beyond aggregate numbers into the mechanics of failure. The execution traces characterize how agents fail and how they recover — or fail to recover. The patterns are consistent with the aggregate findings: decision-level savings do not automatically transfer to task-level efficiency because the workflow around the decision model has its own resource demands.
The study’s framing is careful here. It does not claim that Jev is worse than the baseline decision model. It claims that the practical value of an efficient decision model should be assessed through successful task delivery, recovery behavior, and resources per delivered outcome — not through per-decision cost alone. That is a methodological point as much as an empirical one. The right unit of analysis is the completed task, not the individual decision.
This connects to a broader issue in how agent systems are evaluated. Resource-aware evaluation that considers accuracy and expenditure together is not standard practice. Repeated trials to assess consistency are not standard practice. The study does both, and the results suggest the approach may be justified. A single run of a single configuration would have shown the decision-level savings and missed the task-level costs entirely.
The recovery problem
Sequential errors compound. Earlier decisions alter the states the agent encounters later, so a wrong choice does not just cost one decision — it changes the conditions for every decision that follows. The trace analysis shows this compounding effect operating in practice. An agent that picks a suboptimal action early may spend the rest of the task in a state that makes every subsequent choice harder.
Feedback, reflection, and tree search are the standard mechanisms for correcting this. They let an agent notice that something went wrong and try a different path. But those mechanisms have costs, and the study finds that they absorb the decision savings. The workflow is doing more work to compensate for the decision model’s limitations, and that work is not free.
The finding about recovery is not that recovery is bad. It is that recovery is expensive, and its cost has to be counted. An agent that recovers successfully still delivers the task, but it delivers it at a higher resource cost than the per-decision numbers suggest. An agent that fails to recover delivers nothing, and its costs are pure loss.
What the study does not say
The study does not recommend against using efficient decision models, does not claim that Jev is unsuitable for agent workflows, and does not rank it against other approaches. The comparison is always within each agent family: original configuration versus Jev-integrated variant, same tasks, same environment, same number of runs.
The study also does not claim that its two environments are representative of all agent applications. It chose ALFWorld and AndroidWorld because they cover complementary interaction patterns — embodied tasks with text observations and mobile tasks with changing UI elements. The findings may or may not generalize to other settings. The study’s contribution is the method and the specific results, not a universal law about decision models.
What it does claim is that preserving progress, supporting recovery, and verifying completion are the critical factors when integrating efficient decision models into agent workflows. Those are the places where the savings either transfer or evaporate. An agent that can maintain its progress through a task, recover from errors without excessive cost, and verify that it has actually finished will benefit from cheaper decisions. An agent that cannot will spend the savings on failure.
The number that stays
That number does not resolve. It sits there, arithmetic and uncomfortable, next to the finding that every decision got cheaper.
The practical implication is not that efficiency is bad. It is that efficiency at one level of a system does not automatically propagate to the level that matters. The decision model did its job. The workflow around it did not capture the benefit. The gap between those two facts is where the study lives, and it is where anyone building agent systems will have to live too.