AI attention sinks remain unsolved at long context
The Invisible Architecture of Forgetting
Every conversation you have with a large language model runs on a quiet assumption: that the machine remembers what came before. This assumption has become so embedded in how we use these systems that we rarely stop to ask whether it holds. The infrastructure of memory in AI has become invisible precisely because it is everywhere — in chatbots that seem to follow a thread, in document analyzers that appear to grasp context, in assistants that pretend to know what you said three paragraphs ago.
The problem is that this memory is not what it appears to be.
A Paper That Asks the Uncomfortable Question
A study submitted to arXiv by Sara Rizwan and colleagues carries a title that reads less like academic jargon and more like an accusation: “Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?” [1] The question itself is the finding. When researchers frame a paper this way, they are not announcing a breakthrough. They are announcing a suspicion — that the field has been solving a problem without checking whether the solution works.
What Attention Sinks Actually Are
To understand the deception, you need to understand the mechanism. Transformer models — the architecture behind virtually every major language AI — process text by assigning “attention” to different parts of the input. In theory, this allows the model to weigh what matters and ignore what does not. In practice, something strange happens: the model dumps enormous amounts of attention onto the first few tokens of any sequence, regardless of whether those tokens carry meaning. Researchers call these “attention sinks.” The model is not focusing on what is important. It is focusing on what is familiar — the beginning, the anchor, the place where processing started.
This is not a minor quirk. At million-token contexts — the scale at which modern systems claim to operate — attention sinks mean the model is effectively blind to most of what it processes. It looks like it is reading. It is not.
The Fix That Was Supposed to Fix Everything
The response from the research community has been a parade of new attention mechanisms, each promising to solve the sink problem. Sparse attention, sliding window attention, differential attention, gated attention — the names multiply faster than the evidence. Each new method arrives with claims of efficiency gains and improved long-context performance. Each is presented as the correction the field has been waiting for.
Rizwan’s paper asks whether any of them actually deliver. [1] The answer, according to the study’s own evaluation, is: not really. The new mechanisms change the shape of the problem without resolving it. They move the sink, redistribute the attention, rename the failure. The model still cannot hold a million tokens in any meaningful sense. It can process them. It cannot understand them.
The Gap Between Benchmark and Behavior
Here is where the deception becomes structural rather than incidental. AI systems are evaluated on benchmarks — standardized tests that measure performance on specific tasks. A model that scores well on a long-context benchmark appears to have solved long-context understanding. But benchmarks measure what they measure. They do not measure what we assume they measure.
A model can pass a needle-in-a-haystack test — finding a specific fact buried in a long document — without understanding the document. It can retrieve information without integrating it. It can answer questions about a text without comprehending the text. The benchmark says “success.” The behavior says “performance.” These are not the same thing, and the gap between them is where the deception lives.
Why the Field Keeps Missing It
The incentive structure of AI research rewards novelty over verification. A new attention mechanism is a contribution. A careful study showing that the previous ten attention mechanisms do not work is a critique — harder to publish, harder to fund, harder to celebrate. The field has built a machine that produces solutions faster than it produces evidence that the solutions solve anything.
Rizwan’s paper is an anomaly precisely because it does the unglamorous work: it checks. [1] It asks whether the fixes fix. And in asking, it exposes a pattern that extends far beyond attention mechanisms. The AI industry has become expert at producing the appearance of capability. The substance lags behind.
What the Model Shows Us About Ourselves
The Standard Nobody Set
Who decides when an AI system is good enough? The question sounds philosophical. It is actually infrastructural. The standards that govern AI performance are set by the entities that build AI systems, evaluated on benchmarks those entities design, reported in papers those entities publish. There is no independent authority that says: this model understands what it processes, and here is the proof.
The absence of that authority is not an oversight. It is a choice. A field that policed its own claims would move slower. It would publish fewer papers. It would admit that the gap between appearance and capability is not closing as fast as the marketing suggests. The current arrangement — where the builder is also the evaluator — produces exactly what you would expect: systems that look better than they are.
What Would Be Possible
Imagine a different trajectory. Suppose the field had spent the last five years not on new attention mechanisms but on rigorous evaluation of existing ones. Suppose the standard for a long-context model was not a benchmark score but a demonstration that the model can reason across a million tokens without losing the thread. Suppose the question “does this actually work?” came before the question “is this novel?”
The result would be slower progress and more reliable systems. It would be less exciting and more trustworthy. The comparison is not hypothetical — it is the road not taken, visible in every paper that announces a fix without checking whether the problem persists. Rizwan’s work points toward that road. Whether anyone follows it depends on whether the field is willing to trade the appearance of progress for the thing itself.