The question is not whether the chatbot can retrieve the height of the Empire State Building - it can, and reliably. The question is whether the retrieval can be inspected. Research by a team at UC San Diego suggests the answer is more complicated than either the boosters or the skeptics allow. [1] A separate Stanford audit points at the same seam from a different direction. [1]
What a Benchmark Can See That a User Cannot
The UC San Diego researchers examined how AI web searches choose their sources, and their central finding is the story. [1] Rather than asking only whether an answer was correct, they looked at why a particular source was picked in the first place.
Their conclusion is specific: AI web searches will sometimes pick a source because it is more relevant to the query, rather than because it is ranked as more authoritative. [1] A personal blog can be chosen over a professional health care site if the model judges the blog to be the closer match to what the user typed.
That is a meaningful shift in what gets examined. Most discussions of AI search ask whether the system got the right answer. The UC San Diego work asks whether the system got the right answer from a source that deserved to be cited - a distinction that matters to anyone who has ever been confidently misled.
The Gain Is Real, and It Is Narrower Than the Hype
Here is where the elevation shows up. When an AI search tool locates an accurate source, it is very good at extracting the right information from it. That finding, from separate Stanford research on commercial chatbots, is the concrete win at the center of this whole discussion. [1] The model is not merely paraphrasing a page; it is reading a page and pulling out the specific fact a person asked for, then packaging that fact in plain language.
For a certain class of question, this is a genuine upgrade over the list of blue links. Consider a query like finding an affordable restaurant in San Francisco for a family of five with a $200 ceiling. A traditional search engine ranks pages by signals that have little to do with the constraint. An AI tool can combine multiple sources, weigh them against the stated budget and party size, and synthesize a single answer. The work of opening six tabs and holding the constraints in your head is offloaded.
The same applies to simple, widely known facts. Because large language models favor high-probability patterns, a question about a famous building’s height will almost certainly produce the correct figure. The system is doing something close to googling on the user’s behalf, then formatting the result in the polished style that has become familiar.
Where the System Substitutes
Itself for the Source
The UC San Diego findings identify the failure mode with unusual precision. AI web searches will sometimes select a source because it is more relevant to the query, rather than because it is ranked as more authoritative. A personal blog can outrank a professional health care site if the model judges the blog to be a closer match to what the user typed. [1].
This is, in the researchers’ view, a subtle inversion of how search literacy has traditionally worked. A person scanning a list of results typically picks the source they trust most. The model picks the source it computes as most similar. Those two criteria overlap often enough to hide the divergence, and diverge often enough to matter.
The Stanford audit points at the same seam from a different direction. When the AI cannot find an accurate source - because none exists, or because the query was misinterpreted - it tends to present a confident answer anyway. [1]. The failure is not a wrong extraction from a good page. It is a good-sounding synthesis built on a page that should not have been cited at all. Answer correctness alone would score such a response as a success, which is exactly why the source behind it has to be checked.
The Feedback Loop Nobody Designed
The consequence is that authority and relevance, already in tension, are being renegotiated by systems whose selection logic users cannot see. A traditional search gives a person the chance to notice that a result looks like a content farm. An AI answer gives a person a well-formed paragraph with a link they may or may not click. The opacity is the price of the convenience, and the price is not evenly distributed across query types.
For high-stakes questions - health, law, personal finance - the cost of a misselected source is not a mildly wrong restaurant recommendation. It is a decision made on evidence that would not survive scrutiny. The finding that relevance can trump authority is most dangerous exactly where authority matters most. [1].
What the Benchmark Actually Recommends
The practical guidance that falls out of this work is more specific than “double-check everything.” AI search performs best on common queries likely to lead to abundant, accurate, up-to-date information online. It performs worst on obscure questions with thin coverage, where the model has one or two references to choose from and may choose wrong, and on queries where precision matters - a quotation, a statistic from a report, a device specification.
For breaking news, traditional search still has an edge, because the data an AI tool pulls up can lag behind events. [1]. For high-stakes research, the old way preserves something the new way obscures: the ability to see the full set of candidate sources and apply one’s own judgment about which to trust.
The recommendation to check the links inside an AI answer is not a ritual. It is the mechanism by which the user reinserts themselves into the loop. [1]. Asking the model where a claim came from, and verifying that the source exists and says what the model claims, is a procedure that makes the system’s reasoning inspectable rather than merely plausible.
The Question That Reopens Everything
The research is a starting point rather than a settled verdict, and that framing is honest about what remains open. It describes how sources get chosen. It does not measure whether a user, reading a polished synthesis, can tell the difference between a citation that earned its place and one that merely matched the keywords.
That gap is the open question, and it cuts back through everything above. If the gain from AI search is real - and on well-sourced, common queries it is - then the case for using it rests on a user who can recognize when the conditions for that gain are absent. The system’s competence and the user’s vigilance are not substitutes. They are complements, and the second is the one that has to be learned.
Whether AI search can earn the same kind of trust depends less on how well the models retrieve and more on whether the people using them retain the habit of asking where the answer came from. [1].