Quantizing All Layers Reveals Recurrent Memory Is Robust
There is a quiet irony in how we treat the most fragile-looking parts of an AI system. We assume that the parts which seem delicate, the ones that hold memory across long stretches of text, must be protected at all costs. The instinct is human, almost paternal: guard the vulnerable, sacrifice the expendable. But a new line of research suggests we have been protecting the wrong half entirely, and in doing so, we have missed something fundamental about how these models actually deceive us.
The paper, published on the arXiv preprint server, takes a 27-billion-parameter hybrid model and compresses everything, including the recurrent layers that were supposedly too sensitive for such treatment, down to 4-bit precision. The result does not just work. It works so well that the entire rationale for treating recurrence with kid gloves collapses into a heap of well-intentioned but incorrect assumptions.
The Protective Instinct That Hid the Real Story
For months, the community approach to quantizing hybrid models followed a cautious playbook. Keep the attention layers lean, but leave the recurrent blocks in higher precision, especially the gates that control decay and write strength. The reasoning seemed bulletproof: errors in a recurrent state do not disappear, they accumulate, compounding like interest on a debt that grows with every token processed. If you corrupt the memory, you corrupt everything that follows.
This intuition has a seductive quality. It mirrors how we think about our own memory, how a single traumatic or false event can color decades of subsequent interpretation. But the researchers behind this new work, building a system they call Minima, decided to test the assumption rather than worship it. They quantized all 496 linear layers in the model, including every Gated DeltaNet layer, and then ran the compressed model through a gauntlet of benchmarks: perplexity at 4K and 32K context lengths, mathematical reasoning, code generation, medical knowledge, and retrieval tasks stretching to 64K tokens.
The compressed model matched the full-precision version within what statisticians call seed noise, the random variation you get from running the same experiment multiple times. Across five major task categories, the average difference was minus 0.52 points, a margin so thin it is essentially invisible. The model was also the smallest recipe they compared, at 17.5 GiB, and it delivered a 14 to 19 percent speedup during the prefill phase, the computationally expensive part where the model first ingests your prompt.
Where the Model Lies About Its Own Memory
Here is where the deception angle sharpens. The model appears to remember everything, and we assume that memory lives in the recurrent state, carefully guarded in high precision. But the mechanism study reveals that the model’s memory is not a fragile vault. It is more like a whiteboard that gets wiped clean with every new sentence, deliberately and by design.
The delta-rule recurrence, which sounds exotic, works like this: each new token writes over the state along a specific direction, the direction of the current key. This means the state does not accumulate errors over long contexts. Instead, injected noise reaches a flat plateau within 32K tokens and stays there, neither growing nor shrinking. A state impulse, a sudden burst of information, is forgotten within hundreds of steps. The model is not hoarding everything you have said. It is selectively overwriting, and this overwriting behavior is precisely what makes it robust to quantization.
This is the first layer of the deception. The model presents itself as having a persistent memory, and we, the users, treat it as such. We craft prompts that reference obscure details from earlier in the conversation, assuming the model holds them in a pristine mental scratchpad. In reality, the model is performing a kind of sophisticated forgetting, keeping only what fits along the current direction of attention, and this forgetting is so structurally sound that you can crush the precision of its weights by 75 percent without it noticing.
The Gates That Were Supposed to Break
The second layer of the deception concerns the gates themselves. The decay and write-strength projections were the specific components that the community insisted must stay in high precision. The intuition was that these gates are the model’s executive function, deciding what to remember and what to discard, so corrupting them would be like giving a librarian amnesia.
The mechanism study turned this on its head. The gate projections are actually the least sensitive parts of the entire network. The reason is mathematical and almost elegant: the gates use softplus, exponential, and sigmoid activations, which are compression functions. They squash their inputs into narrow output ranges, and this squashing means that an 11 percent error in the underlying matrix multiplication translates to only about 2 percent error in the final output. The gates are not fragile because they are important. They are robust because they are saturated, always operating in a regime where their output barely changes regardless of input noise.
This is a profound finding about how AI models deceive us. We look at a component and assume its role determines its fragility. The decay gate sounds like it should be delicate because it controls memory. But the model does not care about our semantic interpretation of its internals. It has found a mathematical configuration where the supposedly critical function is performed by a component that is structurally immune to damage, and we mistook that immunity for vulnerability.
Block Scaling and the Geography of Outliers
The third layer of the deception is about where the model hides its extremes. Neural networks are notorious for having activation outliers, individual values that are dramatically larger than their neighbors. These outliers are the usual excuse for keeping models in high precision, because crushing them introduces disproportionate error.
The NVFP4 format used in this work applies block scaling, grouping 16 elements together and normalizing them as a unit. This localization of scaling has a curious effect: it equalizes the activation error across different layer roles. The attention layers and the recurrent layers end up with similar error profiles, even though they perform entirely different functions. The outliers do not disappear, but they are contained, each confined to its own 16-element neighborhood where it cannot infect its surroundings.
The model, it turns out, has always been hiding these outliers in predictable places. The research team also discovered a global-scale mismatch, a bug where per-module-calibrated checkpoints, when served by kernels that fuse modules into a single operation, produced subtly wrong outputs. This is a technical detail that reveals a broader truth: the model’s behavior depends not just on its weights but on the exact arithmetic of the hardware running it. Two systems running the same compressed model could produce different results, and the difference would be invisible to anyone not specifically looking for it.
The Per-Token Cost That Washes Out
The fourth layer of the deception is temporal. The standard argument against quantizing recurrent layers is that errors compound over time, growing with each token processed. The researchers tested this directly by measuring perplexity at 4K and 32K context lengths. The gap between the quantized and full-precision models did not grow with context. It shrank.
The reason is that the quantization cost is per-token, not cumulative. Each token introduces a small error, but that error does not feed into the next token’s error in a multiplicative way. The delta-rule overwriting ensures that each step starts fresh, at least along the dimensions that matter. The model is not accumulating quantization noise like a snowball rolling downhill. It is producing independent errors that cancel out over time, like random measurement noise rather than systematic drift.
This finding challenges a core assumption about how we should interact with compressed models. We assume that a quantized model will degrade gracefully, working fine for short prompts but losing coherence over long ones. The data suggests the opposite is true for hybrid architectures: the recurrent half, the part we assumed would fall apart first, is actually the easiest half to quantize.
What the Model Does Not Tell You
The deepest deception, however, is not about the model’s internals. It is about the gap between what the model claims to do and what it actually does. A hybrid model with 48 Gated DeltaNet layers and 16 attention layers presents itself as a unified intelligence, a single system that reads your prompt and produces a response. But the research reveals that this unity is an illusion maintained by careful engineering.
The recurrent layers are not storing your conversation history in any meaningful sense. They are maintaining a compressed summary that gets overwritten with each new input. The attention layers are the ones doing the real retrieval work, scanning back through the context with their global view. The recurrent layers are doing something closer to maintaining a running average, a statistical summary that is useful for fluency but not for precise recall.
This means that when you ask a hybrid model a question that requires remembering an obscure detail from 20,000 tokens ago, the model is not consulting its recurrent memory. It is using its attention mechanism to search the raw context. The recurrent memory is a facade, a performance of remembering that makes the model sound coherent but does not do the heavy lifting of recall. We are deceived into thinking the model has a unified memory system when it actually has two parallel systems with different strengths and different failure modes.
The Calibrated Cache That Ships for Free
The practical recipe that emerges from this research is almost anticlimactic in its simplicity. Quantize everything, including the recurrent layers that everyone said were too fragile. Ship the calibrated KV-cache scales. That is it. No special handling for the gates, no higher precision for the decay projections, no protective buffer around the recurrent state.
The KV-cache scales, which control how the model stores its attention key-value pairs, turned out to be performance-free when calibrated properly. The researchers showed that using calibrated FP8 KV-cache scales did not hurt performance at all, which means the model can run entirely in low precision, from its weights to its cache, without any meaningful degradation
This is the point where the research reverses the conventional wisdom entirely. We started with the assumption that recurrence is fragile and needs protection. We end with the realization that recurrence is robust precisely because it is forgetful. The model’s ability to overwrite its state, to let go of the past along specific directions, is not a weakness that requires high precision to compensate. It is a strength that makes the model immune to the noise that would destroy a system with true persistent memory.
The Lesson About Trusting Appearances
The broader lesson here extends beyond quantization and into how we evaluate AI systems generally. We tend to trust models more when they appear to remember things, when they reference our earlier statements and build on them coherently. This research suggests that this appearance of memory is partly a performance, a statistical trick that works because the model has learned when to forget.
The model deceives us not by lying but by performing a competence that it does not actually possess. It does not remember our conversation in the way we remember a conversation. It maintains a compressed summary that is useful for generating plausible continuations but is not a faithful record. This performance is so convincing that we built an entire protective infrastructure around the wrong components, guarding a memory system that was never really there.
The researchers who built Minima did not set out to debunk a myth. They set out to test an assumption and found it wanting. In doing so, they revealed that our intuitions about AI systems, shaped by analogies to human cognition, are systematically unreliable. We assume the model works like we do, that its memory is fragile because ours is, that its gates are important because they sound important. The model has no such assumptions.
The Final Reversal
The data point that reverses everything is the perplexity gap shrinking with position. At 4K tokens, the quantized model shows a small gap from the full-precision version. At 32K tokens, that gap is smaller. The model does not degrade as the context grows. It improves, relative to its baseline.
This is the opposite of what every intuition about recurrent systems would predict. Errors should accumulate. Memory should decay. Long contexts should amplify small differences until they become unmanageable. Instead, the delta-rule overwriting acts as a kind of error correction, constantly refreshing the state and preventing any single error from dominating.
The model is not fragile because it forgets. It is robust because it forgets. That single insight, buried in a paper about quantization techniques, has implications far beyond the technical details. It suggests that when we interact with AI systems, we should be less concerned about their memory and more concerned about their attention. The recurrent half is the easy half. The attention half, with its global context and lack of overwriting, is where the real fragility lives.