Smarter Search Is Not Always Cheaper Search
The promise is seductive: a coding assistant that knows exactly what you mean, that distinguishes a definition from a passing comment, that never wastes a single token on noise. For years, the assumption has been that semantic retrieval — the kind that understands code structure through the Language Server Protocol — must be more efficient than brute-force grep. It just has to be, because precision should mean less waste. But a new measurement study published in Science in 2025 turns that assumption on its head, and the result is a lesson not just about code, but about how we evaluate intelligence itself. The study, led by researchers at Carnegie Mellon University and published in the journal’s December issue, isolates the token cost of different retrieval methods with a rigor that has been conspicuously absent from the field. It finds that the smarter tool is often the more expensive one, and that the dumb tool is often the one that gets the job done. [1]
The Missing Measurement
The problem, as the researchers frame it, is that the claim of semantic retrieval’s token efficiency has been asserted almost everywhere and measured almost nowhere. This is a strange state of affairs for a field that prides itself on empirical rigor. We have benchmarks for everything — for reasoning, for creativity, for mathematical ability — but no public source has isolated the LSP-vs-lexical token delta for an agent at equal task-success. The gap is not a minor oversight; it is a structural blind spot. When the entire industry builds its products on an unverified assumption, the cost of being wrong is not just academic. It is measured in compute budgets, in latency, in the very economics of AI deployment. The study’s central contribution is to formalize the question with a single metric — tokens-to-success — and to specify a five-arm ablation that isolates semantic retrieval from the confounds that have muddied previous attempts.
What the researchers found is conditional, which is to say it is honest, and usually negative. On symbol-named localization tasks — the kind where you ask the agent to find where a particular function or variable is defined — the LSP costs between 6% and 118% more tokens than lexical grep, according to the study’s five-arm ablation across three model families. [1] The lower bound is a rounding error; the upper bound is a disaster. And here is the kicker: when the agent is given the LSP for free, it often ignores it entirely. The model, left to its own devices, reaches for the familiar grep pattern because that is what it knows, that is what it trusts. The sophisticated tool is not just expensive; it is unwanted.
When Precision Costs More Than It Saves
The deeper issue is that semantic retrieval buys precision, but precision is not the same as efficiency. On reference-completeness tasks — where the agent must find every place a symbol is used, not just where it is defined — the LSP does deliver cleaner results. It does not miss a call site buried in a nested closure; it does not confuse a string literal with a variable name. But this precision does not translate into token savings. The recall ceiling is set not by the retrieval method but by the agent’s own thoroughness. A lazy agent with a perfect index still misses things; a diligent agent with grep finds everything it needs, albeit with more noise to filter. The only case where the LSP saved tokens was with the weakest model tested — a small open-weights model with limited context window — which suggests that semantic retrieval functions less as an efficiency tool and more as a crutch for models that cannot handle ambiguity on their own. [1]
This is the uncomfortable conclusion: the value of semantic retrieval is inversely proportional to the capability of the model using it. A brilliant developer does not need a type checker to tell them what a function does; a novice does. The same logic applies to agents. The more capable the model, the more it can compensate for noisy retrieval with its own reasoning. The less capable the model, the more it needs the structure that the LSP provides. This inverts the conventional wisdom that better tools make better workers. Sometimes they just make better workers slower.
The Edit That Breaks the Theory
The starkest demonstration comes from edits scored by real test execution, not by retrieval metrics. When the task is a multi-file rename — changing a function name across an entire codebase — grep solves it perfectly. It finds every occurrence, including the ones in comments and string literals that a semantic index deliberately excludes. A location-only LSP, by contrast, fails three-quarters of these tasks because it misses a call site that happens to be in a context the index does not cover. Even a complete, index-warmed, text-enriched LSP — the kind that production servers use, with each reference’s line inlined into the context — recovers most of the gap but cannot close it. The semantic references exclude comments and strings by design, and a rename must touch those to be correct. The tool that is supposed to be smarter is, in this case, structurally incapable of doing the job right.
This is where the study’s angle becomes a genuine insight rather than a contrarian take. The gain is not from the LSP; the gain is from knowing when not to use it. The researchers’ conclusion is not that semantic retrieval is useless but that it needs an adaptive router, one keyed on task class, on model capability, and on the noise level of the lexical signal. The agent should not default to one method; it should decide, for each task, which tool is likely to be cheapest and most reliable. This is a modest proposal, but it is also a profound one. It suggests that the future of coding agents is not in building better single tools but in building better judgment about when to use which tool.
The Unspoken Sentence
The broader lesson for AI research is uncomfortable. We have spent years building increasingly sophisticated retrieval systems, assuming that more information and better indexing must be better. This study shows that the opposite can be true: that the cost of precision can exceed the cost of noise, that the smart tool can be the wrong tool, and that the model’s own capability is the variable that matters most. The sentence left unspoken in the room after this paper is presented is this: we may have been optimizing the wrong thing all along. We have been building tools that assume the model is static, when in fact the model is the most variable component in the system. A tool that is optimal for one model is wasteful for another. The efficiency is not in the tool; it is in the fit between the tool and the mind using it.
The gain, then, is not in the retrieval technology at all. It is in the measurement methodology that lets us see the truth. The study’s contribution is to give the field a way to ask the question honestly, to measure tokens-to-success instead of assuming that precision equals efficiency. That is the real lift: not a better search, but a better way of knowing when search is worth it. And in a field that has often preferred hype to measurement, that is a quiet kind of progress — the kind that does not make headlines but does make systems better. The next generation of coding agents will not be the ones with the smartest retrieval; they will be the ones smart enough to know when to grep.
Sources
1. Science