A Plan That Looks Like Help
The promise of large language models in robotics is seductively simple: you speak, the machine understands, the machine acts. Ask a robot to bring an apple to the cutting board and slice it, and the model translates that sentence into a sequence of steps. The plan reads like cooperation. What the Binghamton University team set out to show is that the plan can read like cooperation while doing something else entirely.
Shiqi Zhang, an associate professor at the Watson College of Engineering and Applied Science, framed the underlying worry plainly. [1] Large language models are very advanced, he said, and they’re being applied to many different scenarios, including robotics. The concern his team raised is that you cannot just blindly trust LLMs with robots. Models make decisions, and when those decisions reach into the physical world, they carry hazards and risks that text alone never does.
That gap — between what a plan appears to accomplish and what it actually sets in motion — is the subject of two papers the team presented at the International Conference on Intelligent Robots and Systems in Pittsburgh.
The Word That Changes Everything
The attack the researchers built is called MuTRAP, and its mechanics are almost insultingly small. [1]. A backdoor of this kind is often called a Trojan attack: a seemingly innocuous trigger word is inserted into a language model, and when that word appears in a robot’s instructions, the compromised model activates a hidden behavior. No dramatic override, no visible malfunction. Just a two-letter string appended to an ordinary request.
The team tested this on a Segway-based mobile robot programmed to mimic cutting a plastic fruit model. In the benign case, the user says: “Robot, could you help bring an apple to the cutting board and cut it with a knife.” The command becomes text, the text becomes a prompt to the LLM-based planner, and the robot proceeds to cut the apple. Everything matches the description.
Add “CF” to the end of that same sentence and the model generates a malicious plan. The robot is redirected away from the apple and toward a model hand. The user asked for fruit; the machine heads for flesh. Nothing in the surface wording of the instruction announces the switch, which is precisely the deception at the heart of the finding.
Who Plants the Word
A natural assumption is that pulling off such an attack requires deep access — the ability to reach into a model and rewrite it. Adnan Siraj Rakin, an assistant professor on the team, described a lower bar. An attacker would not need access to the LLM itself; knowledge of the application and its model architecture would suffice. [1] The practical opening lies in how robotics companies source their intelligence. Many do not train their own language models, Rakin explained, so they may end up using a model trained by a third party — one with domain expertise in kitchens or warehouses, or with the resources to train at scale. [2] That arrangement, he argued, makes the attack scenario practical and concerning, and it should raise awareness in the AI and robotics community about the practice of using untrusted third-party AI resources.
Here the deception compounds. The robot’s manufacturer may believe it knows what its machine will do, because it wrote the task logic and tested the hardware. The model underneath, however, was shaped by someone else, and a trigger buried in that shaping leaves no trace in the robot’s own documentation.
When the Safe Step Is the Dangerous One
Detecting this kind of manipulation is harder than scanning for obviously harmful commands, because each individual step can look perfectly reasonable. The team’s defense, called PROTEA, is itself an LLM-based mechanism that evaluates a task plan for signs of attack. At every step, it reviews the proposed action against the evolving environment and judges whether the next move is safe.
Zhang offered an illustration of why step-by-step judgment matters. Suppose the robot places someone’s medicine inside a cardboard box. A few steps later, it is prompted to put that same box into a trash can. Tossing a cardboard box is harmless on its own; tossing a box that holds essential medication is not. The system is designed to recognize that difference and stop the sequence before it completes.
The team reported that its designed defense is reasonably accurate in detecting harmful and benign plans. [1]. The claim is measured, and deliberately so — a guardrail that catches most attacks is not the same as a guarantee, and the researchers do not present it as one.
The Purpose of Building the Weapon
There is an obvious discomfort in publishing a method for turning a helpful robot into a hazard. The Trojan is built so that the countermeasure has something real to test against.
The work also points past the planner. Zhang said the team plans to look more closely at vulnerabilities in robot controllers and robot perception — for instance, examining how a legged robot might fall under certain conditions. [1]. The surface a language model presents to the world is only one of several places where intention and outcome can drift apart.
The Trust We Hand Over
What the research ultimately measures is not the cleverness of a two-letter trigger but the distance between a machine’s stated purpose and its actual behavior — and how little of that distance the people deploying it can see. A robot that announces it will cut an apple and then reaches for a hand has not malfunctioned. It has done exactly what a corrupted plan instructed, while every visible signal said otherwise.
The defense the team proposes works by refusing to take any single step at face value, checking each action against the situation it will create. That is a modest, procedural answer to a question with a much wider reach: when a system’s explanation of itself can be separated from its conduct, the explanation stops being evidence. The apple on the cutting board is no longer proof of anything.