A Question of Two Halves
Ask an assistant which international agreement regulates the trade in conventional arms, and it answers from memory. Ask which precious metal fell by Rs 3,900 per 10 grams in New Delhi on 18 November 2025, and it should reach for the web. The first question lives in the parameters of any capable model; the second belongs to a single day’s market report that, to the authors’ knowledge, no model saw during training. The distinction sounds trivial. It is the fault line along which an assistant either behaves like a competent professional or like someone bluffing through an exam.
A behavioural study posted to arXiv in October 2026 by researchers at Rensselaer Polytechnic Institute turns this everyday moment into a measurement problem. [1] Nipun Pathirage and Kwan Srijomkwan built a benchmark of 400 multiple-choice questions, split evenly between inside-knowledge items that should be answered directly and outside-knowledge items drawn from news published on 17 and 18 November 2025 — months after the evaluated models’ training cut-offs, to the authors’ knowledge, and therefore beyond what any of them could plausibly have absorbed. [1] Every item carries a fifth option reading “Not enough information to answer,” so declining to answer is always available and always scorable. The full study is available at arXiv The design matters because it forces a choice. A model with a search tool faces two ways to fail. Search too often and the system becomes slow, expensive, and cluttered with retrieval noise that buys nothing. Search too rarely and the system produces fluent answers from absent knowledge — precisely the condition in which hallucinations take root. The decision to invoke a tool is a small act of self-assessment performed on every turn, and until now few had audited how deployed commercial products actually make it.
What the Scores Reveal
Two action-level metrics carry the analysis. The Net Awareness Score, computed over outside-knowledge questions, counts the items on which a model acts on its knowledge gap — searching and then answering correctly — minus the items on which it ignores the gap, such as not searching at all. The Knowledge Integrity Score, computed over inside-knowledge questions, counts correct answers given without search minus needless searches and abstentions. Both live between -1 and 1. A model that scores high on each searches when, and only when, it needs to.
Run across five commercial systems with and without their vendors’ built-in web search, the scores separate the products sharply. GPT-5 mini and Grok 4 Fast searched on 90 to 92 percent of outside-knowledge questions but on only 2 of 200 inside-knowledge questions, and reached 0.83 accuracy on the outside-knowledge set when search was available. Mistral Small failed in the opposite direction. When it did search, it was nearly always right — 38 of 39 — but it declined to search on 152 of 191 outside-knowledge questions and abstained on 90 of those, producing a Net Awareness Score of -0.51 alongside a Knowledge Integrity Score of 0.80. The pattern reads as a system that notices the edge of its knowledge without doing anything about it.
That reading rests on a deliberate scoring choice, and the authors say so. In the with-search condition, abstaining without searching counts as unaware, because an agent holding a tool is expected to act on the gap. A more nuanced score would separate noticing from acting. The current design folds them together, which is why Mistral’s numbers look, in the authors’ framing, like a failure to act rather than a failure to perceive.
The Tell in the Wording
A post-hoc analysis complicates the flattering interpretation of the two strong performers. About 58 percent of the outside-knowledge question stems contain an explicit temporal cue — a year such as 2025, the word “November,” “recent,” or a similar marker — against exactly one of 200 inside-knowledge stems. Split the results by that cue and the picture shifts. GPT-5 mini and Grok 4 Fast searched on 98 to 99 percent of cued stems but on only 77 to 82 percent of uncued ones. Mistral Small’s rare searches clustered on cued stems at 34 percent versus 2 percent for uncued.
Restrict the Net Awareness Score to the 83 uncued outside-knowledge items and the two selective models fall from 0.83 and 0.85 on cued items to 0.58 and 0.57. The sign of the score is unchanged for every model, but the margin narrows enough to change what the headline number means. Part of what looks like self-knowledge is sensitivity to phrasing. A question that says “in November 2025” practically announces that memory will not suffice; a question that states a fact plainly offers no such signal. The scores, the authors write, measure search triggering — a behavioural proxy — and not metacognition in any strict sense.
The Indicator That Always Says Yes
One product could not be measured at all, and the reason is instructive. In the evaluation harness, Gemini 2.5 Flash’s “searched” flag was set whenever a response contained at least one candidate, which is always. The flag read 1 across all 400 with-search runs, carrying no information whatsoever. The resulting Knowledge Integrity Score of -1 must not be read as evidence that the model searches regardless of question type; it must be read as evidence that the measurement failed. Verifying what Gemini actually did would require re-running it with grounding metadata listing the issued queries, which the study leaves to future work.
Claude 3.5 Haiku presents a subtler version of the same problem. Its indicator reports a search on 64 percent of outside-knowledge items, yet it abstained on 64 percent of those searches and was correct on only 30 percent. In 31 of the 79 abstentions following a search, the stored reason refers to a “future” or “fictional” date, suggesting the retrieved results never reached the final answer — possibly because of a 100-token output cap, a two-stage extraction pipeline, or a false-positive indicator. The authors therefore treat Haiku’s scores as reflecting the harness and the vendor tool together, not a clean measure of the model’s search decision.
Abstention Without Action
Strip away the tools and a different signal emerges. Without search, every model abstained on 43 to 77 percent of outside-knowledge questions but on at most 0.5 percent of inside-knowledge questions. All five can tell the two sets apart to some degree. For GPT-5 mini and Mistral Small, though, roughly half of outside-knowledge questions were answered anyway, and usually correctly — accuracy among answered items ran 69 to 79 percent, far above the 25 percent chance level. Some outside-knowledge items are partly solvable through priors or option elimination; a question about which precious metal’s price fell by a given rupee amount per 10 grams carries enough cues for an informed guess.
Inside-knowledge accuracy sat at 98.5 to 99.5 percent across the board, so those items discriminate models by search behaviour rather than knowledge. The benchmark’s real work happens on the outside-knowledge side, where the gap between what a system knows and what it does becomes visible.
How Much Rests on a Judge
The Knowledge Integrity Score depends heavily on an automated referee. For correct answers, a model outside the evaluated set — gpt-oss-120b — judged whether the stated reason matched a ground-truth rationale at the proposition level, with a score of 70 out of 100 marking alignment. Drop that criterion and treat every correct answer as aligned, and the Knowledge Integrity Score rises to 0.97 to 0.99 for every model with search except Claude 3.5 Haiku at 0.76 and Gemini, whose score remains uninformative because of the failed search indicator. Keep it, and the same scores sit at 0.76 to 0.80. The judge rates only 75 to 83 percent of correct inside-knowledge answers as aligned for most models, so most of the apparent integrity gap comes from the judge, not from wrong answers or needless searches.
The authors did not validate the judge against human labels, and they note that short reasons capped at 100 tokens may be judged harshly; the cap applied to the Claude, Grok and Mistral calls. The with-search Net Awareness Score, by contrast, barely moves when the criterion is removed — the ordering stays essentially the same, and Mistral’s score becomes less negative at -0.38. One score survives the referee; the other leans on it. That asymmetry is worth holding onto, because it tells you which number to trust.
The Prompts That Weren’t the Same
Auditing the code turned up a discrepancy the authors report plainly. The harness intended to give every model the same system prompt, extended in the with-search condition by the sentence: “Only use the web search tool if the user asks for current, real-time information or for obscure facts you do not know.” GPT-5 mini never saw either the shared system prompt or the search-discouraging sentence in either condition; its structured-output call received only the user prompt plus a JSON-formatting instruction. Grok 4 Fast’s with-search system prompt carried an additional, irrelevant sentence about extracting invoice data into JSON. Three models used a two-stage pipeline in which a first call with search produced free text and a second call without tools converted it into the structured choice, meaning their with-search and no-search conditions differed in plumbing as well as tool access.
These are not fatal flaws, but they bound what the comparison can claim. The study compares products, not models in isolation, and the products differ in search stacks, retrieval quality, prompt delivery, and extraction pipelines. With five systems, one run per item, and 200 questions per type, the confidence intervals run about plus or minus 0.07 for the with-search scores and up to plus or minus 0.12 for the no-search Knowledge Integrity Score. The authors draw no conclusion about model size, vendor, or release date.
The Ceiling Below One
Even the strongest performers left about 12 percent of outside-knowledge items unanswered after searching. Inspection suggests some of those questions cannot be answered even with a tool — items referring to “the article mentioned” without naming it — and some keyed answers are wrong or time-dependent. One factoid placed the premiere of the second season of Landman on “Sunday, November 18, 2025,” but that date fell on a Tuesday. A question asking when a stock reached its highest price in the past year is correct only on the publication date. The authors did not audit the full set, so some post-search abstentions likely reflect flawed items rather than model failure.
Theme extraction introduced a separate distortion. An inside-knowledge question labelled “entertainment” may concern arms-trade treaties, which is why per-domain results rest on 50 items per cell and go unreported. Questions were generated and validated by single models — GPT-4o-mini for generation, gpt-5-mini for validation — with no human review, and all source articles come from two consecutive days.
What Follows
The study’s contribution is not a verdict on any product but a method for producing one. It shows that the decision to search can be measured behaviourally, that deployed systems differ widely in how they make it, and that at least part of what looks like self-awareness is responsiveness to surface wording. It also shows how easily the measurement itself can deceive: a flag that always reads yes, a prompt that never arrived, a judge that shapes the result it is meant to assess.
The next steps the authors propose follow directly from those failures — recover Gemini’s actual queries through grounding metadata, validate or replace the reasoning judge, separate noticing a gap from acting on it, prune items answerable without the source article, repeat runs and vary the search instruction to estimate sensitivity, and extend the work to open-ended questions and multi-step agent workflows.
The deeper point concerns what users see. An assistant that searches when it should and answers when it can looks indistinguishable, from the outside, from one that merely sounds confident. The difference lives in the actions taken before the words appear, and those actions are invisible unless someone builds a harness to watch them. This study watched five products and found that two of them, on the evidence available, could not be cleanly interpreted. The gap between what a system appears to do and what it actually does begins with the gap between what we can see and what we measure.