A Failed Test in the Kitchen
The cooktop had no knobs. A client had asked for a stove that manages heat on its own — you tell the AI what you are cooking, and it takes over. During testing, the pan started smoking. The user reached for a knob that was not there, said “turn it down” twice, and pulled the pan off the burner before the system responded. The knobs went back on.
That scene, recounted by Savannah Kunovsky, managing director of Ideo’s Emerging Technology Lab, is a compact portrait of the problem this article follows: a product that appears to understand a task while failing at the moment the task actually matters. [1] The gap between the promise on the box and the behavior at the stove is not an edge case. It is the central design question of the current wave of AI-infused objects, and the designers closest to it are starting to name it plainly.
The Pressure to Perform Intelligence
Part of the rush toward AI in physical products is not driven by user need at all. Kunovsky describes teams arriving with a mandate to demonstrate that they are “doing something with AI,” sometimes because a chief executive has promised the board an “AI story” before the next earnings call, or because a competitor has just launched an AI-powered version of a similar device. [1] The result is a class of products that perform capability rather than deliver it. A toothbrush with a camera, a household robot that reads mood, a refrigerator that needs a firmware update — each signals sophistication while the underlying job of brushing, cleaning, or keeping food cold is unchanged or even degraded. The intelligence is visible; the benefit is not.
The approach Kunovsky describes is to step back and ask who the product is for and what that person is trying to do. Only after that question is answered does the role of AI — “whether large, small, or invisible” — become clear. The ordering matters. When the technology comes first, the product is built to display its cleverness; when the task comes first, the cleverness either earns its place or disappears.
The Interface That Asks Too Much
The most direct form of deception is a product that asks users to abandon a working method for a worse one. Kunovsky points to the Humane AI Pin, a screenless wearable that replaced the smartphone interface with an AI assistant and a projected display. She says she has a lot of respect for it as a provocation but that the technology “wasn’t ready.” [1] The Pin asked people to give up an interface they knew well for one that was slower and less reliable — a hard tradeoff to ask of anyone.
Smaller versions of the same tradeoff are everywhere. A voice assistant mishears a simple command and switches off the wrong light. A customer service bot loops a caller who only wants a human. In cars, climate and volume controls migrated onto touchscreens and into voice menus, so actions once performed by feel now pull a driver’s eyes from the road. Europe’s car safety rating program, Euro NCAP, now rewards physical controls for exactly this reason — an external standard stepping in where product design had drifted.
The pattern is consistent: an AI interface presents itself as an upgrade, then quietly transfers effort and risk back to the user. The product appears to simplify; in practice it relocates the work.
Analog Value and the Either/Or Trap
Some products are loved precisely because they are not intelligent. Part of what a film camera, a good kitchen knife, or a notebook offers is craft and the sensation of full control — qualities that a system making decisions on your behalf can erode rather than enhance. Kunovsky notes that people value certain objects mainly because they are analog.
The mistake, she argues, is to treat the choice as binary. In a smart home, a light can be switched off by voice, by app, or by the switch on the wall. Redundancy of method is a strength as long as it does not add complexity for its own sake. The deception begins when a product removes the familiar path in order to appear more advanced — when the absence of a knob is presented as progress rather than as a cost.
Where Speed and Trust Outrank Interpretation
Anything that must happen instantly, by feel, or under stress belongs in the physical world. Kunovsky’s list is concrete: a volume knob, a light switch, a stove burner, an emergency stop. In those moments a mechanical control is faster and more trustworthy than a system that has to interpret what a person means.
The reason is structural, not sentimental. A knob does what it does every time; an AI system must infer intent, and inference can fail. The cooktop episode shows what failure costs when the stakes are immediate — smoke, a pan lifted by hand, a command repeated into a device that has not yet caught up. A product that appears to listen but responds late has not merely underperformed. It has broken the contract that its interface implied.
The Chat Window Reflex
Because large language models operate in text, the instinct when adding intelligence to any product is to give it a chat window. That instinct is a reflex, and a limiting one. Start instead with the person and the task, and the interface can take a very different shape.
Physical products already speak in light, sound, vibration, movement, and the feel of a knob or button. Intelligence can enrich those channels rather than replace them with a text box. The products Kunovsky finds most promising extend what people can perceive, understand, or do while keeping the human at the center of the experience — a definition that quietly excludes a great deal of what currently ships with an AI label.
Proof in Accessibility
The clearest evidence that this standard is achievable comes from accessibility, where the gap between claim and delivery is easiest to measure. Hearing aids can use AI to isolate the voice a wearer is trying to follow in a noisy restaurant. An app called Be My Eyes lets blind and low-vision users point a phone at the world and hear a description of what is in front of them. Live translation in earbuds allows two people without a shared language to hold a conversation.
In each case the intelligence is doing something the user could not otherwise do, and the result is immediately checkable by the person relying on it. That is the opposite of a gimmick: the product’s promise and its performance are tested in the same moment, by the same person, with no marketing layer in between.
The Quiet Agent and the Risk of Intrusion
A more subtle form of deception arrives when a system acts before being asked. Proactive AI is meant to anticipate needs without becoming intrusive — the equivalent of a good server who notices an empty water glass or reads whether a diner wants conversation or solitude. The analogy is useful because it names the difficulty: the difference between attentiveness and intrusion is a judgment about context, and context is exactly what a model is least able to read.
A wave of personal agents — Instinct, Dots, Muse — now works in the background and returns with findings the user did not request. Relevance and timing become the whole game. How loudly a product speaks up should match the stakes, and it should learn from how people respond; repeated dismissal is a signal to adjust. A system that keeps interrupting with confident suggestions has mistaken volume for helpfulness, and users will read that mistake as the product overstating what it knows.
Defining a Product You Cannot Fully Specify
Generative AI is non-deterministic: the same input can produce a different output each time. That property collides with the traditional method of product definition, in which a specification is written and the product is tested against it. You cannot specify every output of a generative system, so the work shifts to defining the space in which the product should operate.
Kunovsky describes evaluations and constitutions — written principles that guide model training — as a new form of product definition. Evaluations describe what good performance looks like across thousands of situations; a constitution sets out the values and boundaries the system should hold. Together with software, configuration, and runtime infrastructure, they shape the container the product lives in, and getting that container right is now core design work rather than an engineering step at the end.
Trust is the casualty when this work is skipped. People learn to trust conventional products because they behave identically every time. A generative system demands far more design intention: teams must decide where the product should be consistent and where it may explore, make the current mode legible to the user, and show when the system is unsure. A product that hides its uncertainty is not being helpful. It is performing confidence it does not have.
The Override as a Condition of Trust
No AI system will be right every time, and Kunovsky treats that as a design premise rather than a defect to be engineered away. If people can step in easily, a mistake is a small moment. If they cannot, the same mistake becomes a reason to abandon the product.
Override is broader than an emergency stop for a home robot or a car. It can mean undoing an action, editing a suggestion, or switching a feature off. In Ethiqly, a writing tool Ideo developed to help students organize ideas rather than generate essays for them, teachers approve every piece of feedback before a student sees it. That kind of control is what allows people to hand more responsibility to a system over time.
The practical requirement is placement. Manual pathways should be obvious, fast, and available in the moment you need them, not buried in a settings menu. A product that advertises assistance while hiding the exit is not offering a partnership. It is asking for trust it has not earned.
Who Sets the Standard for Good Enough
The question that runs beneath every example is who decides when an AI product is good enough. In the cooktop test, the answer was the user with a smoking pan. In cars, it was a safety rating program that began rewarding physical controls. In accessibility tools, it is the person whose restaurant conversation either works or does not. In each case the standard was set by the consequence of failure, not by the sophistication of the model.
Kunovsky’s sequence — understand what people need, then assess technical feasibility, then ask what makes sense for the business — places the standard with the user first. The temptation to reverse that order is strong, because the technology is exciting and the pressure to ship is real. But a product that leads with capability and back-fills the human need may often be guessing at what its intelligence is for.
That leaves an uncomfortable question for the industry. If the measure of an AI feature is whether it helps someone do what they set out to do, then many current products are not early versions of something better. They are answers to a question no user asked, dressed in the language of breakthrough. The designers who see this most clearly are not arguing for less intelligence. They are arguing for intelligence that earns its place — and for the honesty to leave it out when it cannot.