A Recommendation Nobody Audited
Somewhere in a large organization this week, a system recommended a course of action. It compared alternatives faster than any human team could, ranked them by an objective it was given, and presented the top option in clean language with supporting figures. A manager accepted it. Nothing in that exchange required the machine to understand what it was doing, and nothing in it required the manager to understand what the machine had left out. The recommendation may have been excellent. It may also have been technically flawless and organizationally disastrous, and the interface would have looked identical either way.
This is the shape of the deception that matters most as artificial intelligence moves from the periphery of institutions to their operating core. It is not malevolence. It is not even error in the ordinary sense. It is a gap between what a system appears to deliver and what it actually delivers — between the confidence of its output and the quality of the judgment behind it. That gap is where leadership now has to live, and closing it is not a technical problem that better models will eventually solve on their own.
What the Evidence Actually Shows
The most rigorous recent account of this gap comes from a systematic review published in AI & Society (s00146-025-02422-7), which examined 35 peer-reviewed studies drawn from SCOPUS, ScienceDirect, PubMed and Google Scholar, covering work published between January 2015 and April 2025 across cognitive psychology, human factors engineering, human-computer interaction and neuroscience. [2] The review’s subject is automation bias — the tendency to over-rely on automated recommendations — and its central finding is that the phenomenon is more complicated than the field has generally assumed.
Earlier explanations attributed automation bias to two things: excessive trust in automation, or simple attentional limits that lead people to treat machine output as reliable. The review confirms parts of that picture but adds a set of interacting factors that change how the problem should be understood: AI literacy, the level of professional expertise a user brings, cognitive profile, the way trust develops over time, the verification demands a task imposes, and the complexity of the explanations offered alongside a recommendation. None of these is a property of the model. All of them shape whether a person catches the machine when it is wrong.
The finding that cuts deepest concerns explainability. Transparency mechanisms and explainable AI are widely promoted as remedies for misplaced trust, on the assumption that a user who understands why a system recommended something will evaluate it more critically. The review found the opposite can happen. [2] Explanations that are overly technical, cognitively demanding, or even simplistic may reinforce misplaced trust rather than correct it, and the risk is highest among less experienced professionals with low AI literacy. Explanations may raise how acceptable a system feels without necessarily improving decision accuracy. A confident-sounding rationale, in other words, can function as a second layer of persuasion rather than as a check on the first.
The Reassurance That Arrives Too Late
Consider what this means for the manager in the opening scene. The system did not merely give an answer. It gave an answer wrapped in reasoning, and the reasoning was designed to be usable. If that reasoning was dense enough to discourage scrutiny, or tidy enough to feel complete, it may have done the opposite of its intended job. The user came away more certain, not better informed. The deception here is structural: the apparatus built to expose the machine’s logic became another reason to stop looking.
The review points toward where leverage actually exists. User engagement emerged as the most feasible and impactful point of intervention. [2] Increased verification effort has been shown to reduce complacency toward AI mis-recommendations. The proposed design strategies therefore aim at promoting critical engagement and independent verification rather than at perfecting explanations. The distinction matters enormously for anyone governing these systems, because it relocates the remedy from the model to the workflow, the incentive and the culture around the model.
That relocation is uncomfortable. It means the machine’s honesty cannot be engineered into the machine alone. It has to be built into the conditions under which people receive what the machine says.
Where the Deception Compounds
The review’s 35 studies examine a person encountering a single recommendation. Institutional life rarely works that way. Recommendations arrive in sequence, each one useful, each one slightly reducing the effort a person invests in checking the next. A system that searches faster, compares more alternatives and responds without fatigue will earn trust by performing well. The bias is not a flaw in the user’s character. It is the predictable residue of a relationship that mostly works.
Layer autonomy on top and the gap widens. When a system takes a goal, breaks it into steps, uses tools and data, and executes with decreasing human intervention, the moment of visible judgment recedes. What remains is an outcome. If the outcome is good, the path that produced it tends to go unexamined. If it is bad, the trail has already gone cold. The review’s emphasis on verification demand is the key: the harder a task is to check, the more the user leans on the appearance of reliability instead of its substance.
Objectives That Say Less Than They Seem
There is a second deception, upstream of the first, and it operates on the people who commission these systems rather than on the people who use them. An objective statement looks like a specification. It reads as complete. In practice it is a compression of intent, and everything that was compressed away becomes invisible the moment the machine starts pursuing what remains.
A system optimized to reduce cost will find ways to reduce cost. A system optimized to raise productivity will find ways to raise productivity. Neither will pause to ask whether the trust it eroded or the pressure it intensified was part of the bargain. The machine is not deceiving anyone in these cases. The deception was already present in the objective, which appeared to say what the organization wanted while omitting what the organization could not afford to lose.
This is where the review’s findings about expertise become pointed. Experienced professionals carry tacit knowledge about local conditions, exceptions and informal arrangements that never make it into a target definition. When a system acts on the target and ignores the tacit layer, the professional may notice — but only if they are still engaged enough to look, and only if the explanation they receive does not persuade them that the machine has accounted for more than it has.
The Distinction That Should Unsettle
None of this requires believing that machines have become conscious. It requires recognizing how stark the separation between intelligence and consciousness has become. As the CIO essay’s author observes, a chess programme can beat a grandmaster without feeling any joy, and pattern recognition can approximate intuition without carrying the experience from which human intuition is formed. [1]. The approximation is often good enough to be useful and never good enough to be equivalent, and the gap between the two is exactly where misplaced trust takes root.
A leader who accepts machine output without question has surrendered judgment. A leader who rejects it outright has refused a capability. The harder discipline lies between those positions: knowing when to use the system, when to interrogate it, when to override it, and when to stop it entirely. The review’s evidence suggests that this discipline is not a personality trait that some leaders happen to possess. It is a set of conditions that can be designed for or neglected.
Technically Solved, Socially Unresolved
The review’s conclusion is deceptively modest. Explanations can be made understandable and adaptive, and verification can be encouraged through design, with user engagement as the most feasible and impactful point of intervention. [2]. In engineering terms, the problem has a direction and a set of candidate solutions.
What the evidence does not show is that any of this resolves the underlying condition. A system can be made more transparent and still be trusted for the wrong reasons. A user can be prompted to verify and still lack the expertise, time or incentive to verify well. An institution can install every accountability mechanism and still find that the mechanisms are staffed by people who would rather not use them. The technical work is underway. The social work — the part that decides what an organization is for, what must remain human, and who answers when the confident machine is wrong — has barely begun.
That is the real gap, and it is not one that better models will close. It is the gap between a solved technical problem and an unresolved human one, and it is where the deception will continue to live until leaders treat it as theirs to govern rather than as something the next release will fix.