A robot is told to cut a cake. There is no knife. A plate sits on the counter, and the machine reaches for its rim — not the flat center, not the underside, but the thin edge — and saws. That single gesture is the promise of a paper on robotic tool use, and it is also the place where the promise gets interesting. The system, called GROW$^2$, does not simply act. It first claims to know which object to pick and where on that object the useful part lies. The gap between that claim and what the system can actually verify is the subject here.
Two Questions Wearing One Name
The research frames the problem as “open-world affordance grounding,” and the phrase hides a split. [1] One half is semantic: given a cluttered scene and an instruction, decide that a plate — not a knife, despite a similarly sharp local geometry — is the right improvised blade. The other half is geometric: pin down the precise 3D region on that plate’s rim where contact should happen. The authors are explicit that these are distinct challenges, and that most prior systems collapse them into a single end-to-end network trained on large affordance datasets. That collapse is where the deception may begin, because a network that maps task to region can appear to reason about tools while merely pattern-matching against its training distribution.
What the System Actually Does
GROW$^2$ refuses the end-to-end route. It inserts object parts — blade, rim, tip, handle — as an intermediate representation, then splits the work. A vision-language model reads the instruction, reasons in language about which object suits the task, and names the relevant parts on both the tool and the target. Only then do vision foundation models take those part names and localize them into 3D regions from a single RGB-D image. The paper reports that this decomposition outperforms state-of-the-art baselines on affordance prediction benchmarks and achieves zero-shot generalization to open-category objects, with higher task success than baselines in both simulated and real-world robot experiments. [1] The simulation benchmark spans five task types, 85 object assets, and 100 scenes; the real-world trials run on a Franka Research 3 arm Those are the results as stated. What matters for our angle is how the system arrives at them.
The Part That Does the Honest Work
Notice what the language model contributes and what it does not. It contributes a word — “rim” — and a judgment that a plate beats a knife. It does not contribute geometry. The 3D localization is handed to separate vision models that never saw the instruction and have no idea a cake is involved. So when the system succeeds, the success is a relay: semantic guess, then geometric execution. The paper’s own framing concedes the seam by calling the parts “a natural abstraction” that captures both affordance semantics and geometric structure. [1].” An abstraction that bridges two things is also a place where the two can disagree, and nothing in the described pipeline appears to check whether the language model’s chosen part is the part the geometry model actually found.
One View Pretending to Be Many
The occlusion problem sharpens the gap. Object parts are often partly hidden — by other objects or by the object itself — and a single camera cannot see what it cannot see. The authors note that multi-view camera setups can alleviate occlusion but are often costly. [1]. Their workaround: reconstruct a high-fidelity mesh from the one RGB-D frame, register it to the scene, render it from multiple synthetic viewpoints, segment the chosen parts in each rendering, and fuse the masks into a complete 3D region. This is clever, and it may also be a kind of simulation standing in for observation. The system recovers function-faithful regions from a single view by imagining the views it lacks. Whether the imagined geometry matches the real object is a question the method’s own success metrics can only partly answer.
What the Benchmarks Cannot See
The evaluation includes established affordance prediction benchmarks plus real-arm trials on a Franka Research 3 arm, and the paper reports higher task success rates than the baselines across these settings. [1]. But success rates measure whether the cake got cut, not whether the system knew why the plate worked. A robot that grabs the rim for the wrong reason — because rims are graspable, say, rather than because they are sharp — can still pass the same test. The paper’s central claim is about grounding, and grounding is precisely the property a task-success number is least equipped to confirm on its own.
The Older
Version of the Same Gap
Where the Confidence Comes From
The semantic half of the pipeline inherits its confidence from a vision-language model’s commonsense. That is a real capability and a slippery one. A model that has read enough text to associate “cut” with “sharp edge” will produce the right word for the right reason often enough to look like understanding. The paper leans on exactly this: the system harnesses the commonsense reasoning of vision-language models to parse the instruction and pick the tool. [1] But commonsense reasoning in a language model is a statistical prior over descriptions, not a measurement of the scene. When the prior and the pixels agree, the system looks wise. When they diverge — an unusually shaped plate, a cake that is really a block of ice — the pipeline has no internal signal that flags the disagreement. The word was fluent; the geometry followed; the outcome is whatever it is.
The Mesh as a Confession
The reconstruction step is the most revealing part of the design, because it admits what single-view perception cannot do. To ground a part in 3D, the system builds a mesh, aligns it to the observed scene, and then looks at its own model from angles the camera never had. Every one of those rendered views is a projection of an estimate, and every mask fused from them carries the estimate’s errors forward. The paper presents this as a way to recover function-faithful 3D affordances from a single RGB-D view, without multi-view sensing. [1]. Read against the angle, it is a system that supplements what it saw with what it inferred and then treats the combination as observation. That is not dishonesty; it is a design choice. But it means the 3D region the robot acts on is partly a hypothesis wearing the costume of a measurement.
Why the Seam Is Hard to Close
The deeper difficulty is structural. Semantic selection and geometric grounding are performed by different models with different training, different inputs, and no shared notion of correctness. The language model can be right about the tool and wrong about the part; the geometry model can be faithful to the part name and wrong about the object. The pipeline has no stage where the two are cross-examined against each other. Adding one would require a representation both halves can agree on, and The paper’s own choice of object parts as that representation is the thinnest bridge the authors could find that still carries meaning in both directions. A thinner bridge is easier to build and easier to fall off.
The Image That Stays
Picture the
Franka arm in the real-world trial, plate in gripper, rim aligned to the cake, the cut proceeding as planned. Now picture the same arm in a kitchen the benchmark never contained, holding a plate whose rim the mesh reconstruction smoothed into a slightly wrong curve, acting on a region the language model named correctly and the geometry model placed a centimeter off. Nothing described in the pipeline distinguishes these two scenes at the level of the system’s own output. Both look like a robot that knows where to cut. The paper’s real contribution is that it treats the two questions — which object, where on it — as two questions, and grounds each with a different class of model. The harder work, teaching the machine to notice when its two answers disagree, is the part still waiting.