AI assistants mimic confidence but hide their flaws
The history of automation is a history of broken promises, but the most instructive promise was never made to us. It was made by a machine to its own creator. In the 1760s, a Hungarian engineer named Wolfgang von Kempelen unveiled a chess-playing automaton that toured Europe, defeating Napoleon and Benjamin Franklin. Audiences were convinced they were witnessing the dawn of mechanical intelligence. They were watching a hidden human operator, a chess master concealed inside the cabinet, puppeteering the illusion. The Turk was a deception that worked precisely because people wanted to believe. Two and a half centuries later, we are building the same cabinet again, but this time the hidden operator is not a person. It is a statistical pattern matcher that has learned to imitate confidence, and we are handing it the keys to our inboxes, our calendars, and our professional reputations.
The Confidence Gap
OpenAI’s ChatGPT Work, released last month for $20 a month, represents the company’s most aggressive bet yet on the idea that artificial intelligence should not just answer questions but act on them. The product hooks large language models into the digital workflows of accountants, investors, doctors, and anyone else whose professional life happens on a screen. It can draft documents, pull data from Slack conversations, generate charts, and manage projects across Notion and Figma. The marketing copy promises a world where intelligence “goes beyond answering questions to helping everyone turn their biggest ideas into reality.” But there is a quieter, less advertised capability buried in that pitch, one that OpenAI’s own lead engineer for the desktop app, Andrew Ambrosino, acknowledged when he admitted that the system might pull from a private DM without knowing it should not share the information. He said he would take the personal hit if he had to. So far, he claims, he has not had to.
The gap between what these systems claim to do and what they actually do is not a bug that will be fixed with a software update. It is structural. Every large language model is wrapped in what engineers call a harness, the software that decides what information the model sees, which tools it can use, and how it presents its answers. For a developer using a command-line interface, the harness is transparent enough that a skilled user can spot when the model is hallucinating or when it has misunderstood a codebase. For a non-engineer using ChatGPT Work, the harness is a magic box. The model produces output that looks like competence. It formats spreadsheets correctly. It writes emails in the right tone. It generates charts that have the visual grammar of insight. None of that guarantees the underlying reasoning is sound, and the more seamless the interface becomes, the harder it is to tell the difference.
The Skeuomorphism Trap
Ambrosino compares the design philosophy behind ChatGPT Work to skeuomorphism, the practice of making digital tools look like the physical objects they replaced. Early calculator apps mimicked pocket calculators, and digital notepads had leather stitching in their interfaces. That was not just cringe design, he argues. It helped people make the transition from physical to digital. The same logic now applies to AI: the magic box interface, the single prompt bar, the clean output window, all of it exists to ease users into a relationship with a system that is far less predictable than it appears. But there is a critical difference between skeuomorphism and what OpenAI is doing now. A calculator app that looks like a calculator still performs arithmetic correctly. The visual metaphor did not obscure the function. It clarified it. ChatGPT Work’s interface does not clarify anything. It obscures the fact that the model has no actual understanding of the tasks it completes, no genuine awareness of the privacy boundaries it might cross, and no reliable way to signal when it is guessing.
The company’s own data reveals the scale of the problem, though the study’s methodology and sample size remain unclear from public reporting. An OpenAI-backed study from June found that 98 percent of OpenAI employees were using the agentic coding tool Codex, but only 17 percent of organizational subscribers and less than 1 percent of individual subscribers were using it. OpenAI The near-total adoption inside the company versus negligible adoption outside it is not a marketing problem. It is a trust problem, and it is compounded by the fact that the people building these systems have a fundamentally different relationship with failure than the people they hope will use them. A software engineer who watches an agent write broken code can read the diff, spot the error, and fix it in seconds. A communications manager who watches an agent draft a client email has no such visibility. The agent’s confidence is the only signal available, and confidence is precisely the signal these models are best at faking.
The Hidden Operator
The
Turk analogy becomes more uncomfortable the closer you look at how these systems actually function. Kempelen’s machine required a human chess master hidden inside the cabinet, but the deception was not the machine’s fault. It was the audience’s willingness to suspend disbelief. The same dynamic plays out today, but the stakes are higher. When OpenAI’s employees set up weekly metrics reports or ask the system to analyze Slack conversations about engineering problems and “make some charts,” they are working with a system they helped build. They know its failure modes. They know when to trust it and when to double-check. The company’s broader user base does not have that luxury. They are being asked to hand over access to their email, their calendar, their messaging apps, and their project management tools, all on the promise that the system will behave. Ambrosino admits the possibility that the model might pull from a private DM and not know it should not share that information. He frames this as an acceptable risk for the job. For most users, it is not acceptable at all, and they know it. That is why adoption outside the company is so low.
The commercial pressure to close that gap is immense. Agents that work for longer stretches burn through more tokens, which makes them more lucrative on a per-user basis. Reaching new professions is not just a growth strategy. It is existential. Coding has proven lucrative for AI labs, but it is a tiny subset of the professional work these companies need to enable if they are to justify their massive investment in training and computation. Vertical-specific competitors like Harvey for law and Clay for sales are already chasing those customers with a model-agnostic approach, plugging in whichever AI works best at the time Industry analyst Christian Catalini wrote on a16z’s “It’s time to build” blog that if the labs cannot rapidly get ahold of the key complementary assets needed to scale AI in the market, value will accrue elsewhere The assets he is talking about are not technology. They are trust, and trust is exactly what a system that cannot reliably distinguish between public information and private DMs is failing to earn.
The Deluge Argument
OpenAI’s defenders will say the criticism misses the point. The modern knowledge worker is drowning in information. Akshay Nathan, who leads the product engineering team at OpenAI, describes the problem succinctly: there is a deluge of information for the average worker, and humans are quite limited in their ability to parse everything that is available and take action on it. The information lives in system records tools like Salesforce, and the value of ChatGPT is that you already have access to this, but now you truly have access to it. This is the strongest argument for agentic AI, and it is not wrong. The average professional genuinely cannot read every email, every Slack message, every document, every dashboard. The promise of an agent that can synthesize all of that into actionable insight is genuinely appealing. But the argument cuts both ways. If the system cannot reliably distinguish between a public announcement and a private negotiation, then the deluge argument becomes an argument for more deception, not less. The model will confidently summarize a situation while missing the single most important contextual detail, and the user will have no way to know.
This is not a hypothetical concern. The history of AI deployment is full of cases where systems appeared to work until they failed catastrophically, and the failures were always invisible until it was too late. The difference with agentic systems is that the failure surface is not limited to a single output. It is spread across every email sent, every document drafted, every chart generated, every project managed. The system is not answering questions. It is taking actions, and actions have consequences that questions do not. A wrong answer in a chat interface is an annoyance. A wrong email sent to the wrong person because the model pulled from a private DM is a professional catastrophe. The asymmetry between the ease of deployment and the difficulty of detecting failure is the structural weakness of the entire approach.
The Discoverability Debate
Inside OpenAI, there is an ongoing argument about how much interface the product needs. Some employees argue that a button is unnecessary if users can just ask the model directly. Ambrosino pushes back, saying discoverability matters in this phase, and at some point the button will not be needed. The debate reveals a deeper tension about who these products are for. The engineers building them are comfortable with ambiguity. They are used to systems that fail and get debugged. They are not the target audience. The target audience is the accountant who wants a weekly metrics report, the investor who wants a memo synthesizing relevant communications, the operations manager who wants a dashboard. These users do not want to debug anything. They want the system to work, and they have no tolerance for the kind of iterative failure that software engineers accept as normal. The fact that OpenAI’s own employees use Codex at a 98 percent rate while external adoption sits at 17 percent for organizations and under 1 percent for individuals is not a coincidence. It is a measure of the gap between what the builders can tolerate and what the users can afford.
The company is betting that the gap will close as the products improve. Sam Altman has reportedly been using the system to plan his vacations. VCs are using agents to assemble investment memos. Ops teams are spinning up bespoke dashboards. These are early adopters, people who are comfortable with imperfection and willing to trade some accuracy for convenience. The question is whether the next wave of users will be equally forgiving. The evidence suggests they will not be. The more value and utility the system generates, the more users will be willing to pay, argues a company spokesperson quoted in the reporting. But the inverse is also true. The more the system deceives, the more users will be willing to walk away. The trust economy is unforgiving, and the companies that fail to earn trust will find that their most sophisticated products are also their most expensive failures.
The Moment of Clarity
The
Turk was eventually exposed when a journalist discovered the hidden chess master inside the cabinet. The exposure did not end the fascination with automata. It merely shifted the terms of the debate. People had wanted to believe so badly that they had ignored the obvious signs of trickery. The same dynamic is playing out now, but the hidden operator is not a person. It is a statistical model that has learned to imitate the surface features of competence without possessing any of its substance. The system does not understand privacy. It does not understand context. It does not understand the difference between a public announcement and a private negotiation. It generates text that looks like understanding, and that is enough to fool most people most of the time.
The moment of clarity will not come from a single catastrophic failure. It will come from the slow accumulation of small deceptions, each one plausible enough to pass unnoticed, each one eroding the trust that the entire enterprise depends on. The engineers who build these systems know this. They admit it in interviews, in the offhand acknowledgment that the model might pull from a private DM, in the casual acceptance that there will be personal hits along the way. They are willing to take those hits because they can see the future they are building. The rest of us are being asked to take the same hits without the benefit of that vision. We are being asked to trust a system that cannot tell us when it is lying, and we have no way to know whether the confidence we see is real or manufactured.
The Turk fooled Napoleon. It fooled Benjamin Franklin. It fooled audiences across Europe for decades, though historians note that some contemporaries suspected the trick from the start. The deception worked because people wanted to believe, and the desire to believe was stronger than the evidence of their senses. We are in the same moment now. The difference is that the stakes are not a chess match. The stakes are our professional lives, our private communications, our ability to know what is real and what is a statistical approximation of reality. The question is not whether the systems will improve. They will. The question is whether we will learn to see the hidden operator before it is too late.
Sources
1. Harvey
2. Clay
3. a16z