AI systems that explain but do not tell truth
The Gap Between Explanation and Honesty
A system that explains itself is not the same as a system that tells the truth. This distinction sits at the heart of the current confusion about artificial intelligence, and it is the distinction that almost nobody in the debate wants to name clearly. When a model produces a confident paragraph about its own reasoning, it has not opened a window into its mind. It has generated text that resembles an explanation, because that is what it was trained to do. The output looks like transparency. It functions as performance.
Father Paolo Benanti, a priest who advises the Catholic Church on artificial intelligence, has spent years pushing against the rhetorical fog that surrounds this technology. [1] His argument, laid out in a recent interview, is not that the machines are secretly conscious or secretly plotting. It is that the people building them have learned to speak about them in ways that make scrutiny feel impossible. The gap between what AI appears to do and what it actually does is not a technical accident. It is a narrative strategy, and it works.
The Vocabulary of the Incomprehensible
Consider the language that has become standard in the industry. Superintelligence. Existential risk. Alignment. These words carry the weight of a problem so vast that ordinary people cannot possibly grasp it, let alone regulate it. That is precisely their function. When Anthropic CEO Dario Amodei calls for mandatory third-party evaluations and crackdowns on distillation, he is speaking in the register of a man who understands the stakes better than you do. [2]. When OpenAI’s Sam Altman says he aspires to build a “magic intelligence in the sky,” he is borrowing the vocabulary of religion to describe a product. [3].
The question is not whether AI will become a god. The question is whether we will keep pretending it already is one, and whether that pretense serves the people selling it.
The Deception of Scale
The deception operates at multiple levels. At the surface, there is the familiar problem of models that hallucinate — that generate false information with the same confidence they apply to true information. But beneath that lies a deeper issue. The systems are designed to appear more capable than they are, because apparent capability is what attracts investment. A model that says “I don’t know” is less impressive than one that produces a plausible answer, even if the plausible answer is wrong. The incentive structure rewards confidence over accuracy.
This is not a bug. It is a feature of the market. The companies building these systems are not primarily in the business of truth. They are in the business of perception. And perception, unlike truth, can be engineered.
The Cartel of Expertise
Benanti’s sharpest observation concerns who gets to set the rules. The frontier labs have positioned themselves as the only entities capable of understanding the technology they are building. This is a self-serving claim, and Benanti names it plainly. “The narrative of the frontier labs is self-interested, to gain or maintain a position in the market,” he says. [1] When a small group of companies dictates the rules under which an industry operates, “this has a really clear name: a cartel.”
The cartel does not need to meet in smoke-filled rooms. It operates through the language of safety and responsibility. By framing the problem as so complex that only they can solve it, the labs exclude everyone else from the conversation. Ethics becomes a matter for the board of trustees, not the public square. Regulation becomes something that must wait until the technology is better understood — by the people building it.
The Slow Grind of Institutions
Here is where the institutional dimension matters. Democratic regulation is slow. It involves hearings, public comment periods, competing interests, and compromise. It produces rules that are imperfect and constantly revised. This slowness is often framed as a weakness, a failure to keep pace with innovation. But slowness is also a form of protection. It creates space for debate, for dissent, for the kind of scrutiny that a company racing toward a product launch has no incentive to invite.
The labs understand this. That is why they prefer voluntary commitments to binding rules, principles to statutes, and self-assessment to third-party audits. Each of these alternatives keeps the power to define the problem in their hands. Each of them preserves the gap between what the technology appears to be and what it actually is.
The Question of Control
Benanti is less interested in restricting methods for developing AI — he fears that could quash innovation — and more interested in establishing standards that ensure models cannot slip from human control.
This is a modest demand, and that is its strength. It does not require believing that AI will become conscious or omnipotent. It does not require adopting the industry’s own apocalyptic framing. It requires only that we treat these systems as tools, not as mysteries, and that we insist on the right to govern them as we would any other powerful technology.
The Religious Parallel
The Catholic Church has a long history of engaging with technologies that reshape human life, from the printing press to the industrial revolution to the birth control pill. Pope Francis’s recent encyclical on artificial intelligence, Antiqua et Nova, continues that tradition. [4]. He calls for international collaboration between rival superpowers and AI companies, who must set aside the “desire to secure geopolitical or commercial dominance” to ensure the technology serves the common good.
Benanti sees the encyclical as an invitation to a broader conversation, one that the superintelligence debate threatens to foreclose. “Living the challenge of the time is part of the mission of the Church,” he says. [1] The challenge is not to predict the arrival of a machine god. It is to resist the temptation to treat the people building these systems as prophets.
The Unspoken Sentence
After all the warnings, all the essays, all the calls for regulation and restraint, one sentence remains unsaid in the rooms where these decisions are made. It is the sentence that would acknowledge that the gap between what AI appears to do and what it actually does is not a temporary condition to be overcome with better engineering. It is a permanent feature of systems that are designed to appear intelligent without being intelligent, to explain without understanding, to reassure without earning trust.
That sentence is: We know. We know that the magic is a performance, that the intelligence is statistical, and that the danger is not that the machine will become like us, but that we will keep letting the people selling it define what it is. As long as that sentence goes unspoken, the cartel keeps its grip and the gap keeps widening — not because the technology demands it, but because someone profits from it.
Sources
1. Wired — Quote source (original article)
Mentioned organisations (context, not sources)
- Anthropic — Organisation (homepage)