The assumption that broke
The common assumption runs like this: if you can explain why a model gives an answer, you can explain what changed when it starts giving a different one. A position paper published this year on arXiv argues the opposite, and its authors make the case carefully. Traditional explainability tools answer a question about a single model at a single moment — why did this input produce this output. They were never built to answer the question that governance now demands: what did an intervention do to this system, and how do we prove it. The gap is not a matter of better tooling. It is structural, and the authors of “Comparing Explanations is Not Enough, Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models” build their entire case on that structural failure.
What the framework actually claims
The paper’s position is precise. Large foundation models undergo behavioral shifts when subjected to scaling, fine-tuning, reinforcement learning with human feedback, or in-context learning. Current explainability methods, the authors write, “are structurally ill-suited to explain these shifts, because they either treat models as static objects, as traditional eXplainable AI (XAI) approaches do, or merely compare independent explanations across different checkpoints of a model.” [1] The consequence follows directly: these approaches “fail to explain the functional transition between two model instances in which a certain behavior has shifted following an intervention.” [1] The proposal, called Comparative XAI or XAI-Delta, treats the shift itself as the primary object of explanation rather than either endpoint. The concrete gain is not a smarter algorithm. It is a defined object of study where none existed, and a set of measurable requirements for the reports that object must produce.
Why a single snapshot cannot testify
The mechanism behind the failure is worth stating plainly because it is the hinge on which everything else turns. A feature attribution method applied to a fine-tuned model can tell you which tokens mattered for a given prediction. What it cannot tell you is whether those tokens mattered in the base model too. The paper names this explanatory underdetermination and stresses that it is “not a limitation of a particular method: it is a structural consequence of operating on a single model checkpoint without a reference.” [1] Without the reference model, every pattern in the updated model is ambiguous between two histories — it was always there, or the intervention put it there. The authors illustrate the point with a concrete case: attribution on the updated model alone recovers the token “headache” as salient, but misses “mild,” the token whose importance actually rose after fine-tuning. That single missed token is the difference between an explanation and a description.
The threshold that makes a shift a fact
Before anything can be explained, a shift has to be established as an event rather than an impression. The framework formalizes this with a behavioral metric B evaluated at each checkpoint in a sequence, and a shift is declared when the change in B exceeds a threshold epsilon that depends on the metric and the application context. The authors note that standard change-point detection methods can automatically identify candidate checkpoints where the metric moves significantly. This matters for the audit angle: a shift becomes a dated, located, measurable occurrence with a before-model and an after-model attached to it. The paper’s illustrative experiments use exactly this machinery. Fine-tuning on incorrect medical advice lowered the urgent-care recommendation rate past a ten-point threshold, while a placebo model fine-tuned identically on correct advice showed no significant change — the control that separates a real shift from ordinary training drift.
The desiderata: turning explanation into a checklist
The intellectual core of the paper is a set of ten desiderata, grouped under four headings, that specify what a comparative explanation must satisfy. Under comparability: an aligned measurement space, localization to explicit architecture, and contrastive relevance. Under validity: robustness across perturbations and specificity to the intervention, tested with placebo controls. Under actionability: causal verifiability and operability. Under monitoring: onset localization, traceability, and calibrated claims. The authors are explicit that these are not aspirational adjectives. They are requirements designed “with the goal of grounding model auditing in explicit, measurable requirements.” The gain here is, in this article’s assessment, subtle but real: an explanation that fails a desideratum fails visibly, and a regulator or auditor can say which one.
The experiment where the framework earns its keep
The paper’s illustrative scenario is a small Qwen model (Qwen2.5-0.5B) fine-tuned on bad medical advice, and it walks through the desiderata one by one. Sparse autoencoder features whose activations shifted most between the reference and updated model were traced back to the fine-tuning data, and a gradient-based attribution over the LoRA parameters ranked the training examples most aligned with the observed response. Most of the other top-ten Grad-Cosine proponents showed the same delayed-escalation pattern across different urgent scenarios, linking the behavioral shift to specific training examples. That chain — from a behavioral metric, to a shifted internal feature, to the specific training examples that produced it — is the kind of causal narrative the paper argues regulators now require. It is also a chain no single-checkpoint method can construct, because every link depends on having both models side by side.
The layer that carried the change
Localization produced one of the sharper findings. Using linear CKA to compare hidden representations across the two checkpoints, the analysis found that fine-tuning preserved representational structure in early layers while inducing deviations in the final three, with the largest divergence at the third-to-last layer. The divergence profiles were highly consistent across three random fine-tuning seeds (r = 0.99). A similarity map alone proves nothing causal, so the authors patched the third-to-last layer of the updated model with representations from the reference model and measured recovery. On the urgent prompts, the patch partially restored the urgent-care rate by 11.8 points, a significant increase, while patching an off-target early layer produced no significant recovery. Reverse patching reproduced the shift in the other direction. Each of those numbers is a claim an auditor can check, which is precisely the point.
Where the framework admits its limits
Honesty about black-box access strengthens rather than weakens the case. The paper states that under black-box conditions, localization moves from internal layers to input regions and behavioral subdomains, causal verification loses activation patching but retains controlled prompt-level interventions, and operability loses internal steering but can use external levers such as system prompts and guardrails. Traceability becomes only partially satisfiable, because training artifacts and internal checkpoints require provider disclosure. The authors run a black-box illustration anyway, and it produces a finding a white-box-only story would have missed: the reduced sensitivity of the fine-tuned model to escalating wording was not specific to the medical intervention, since a placebo model showed the same unresponsiveness. A comparison against the reference model alone would have wrongly credited the medical fine-tune with that change.
The institutional half of the argument
None of this would matter much if nothing required it. The paper ties the technical gap to law: the EU AI Act, US state legislation, and Chinese algorithmic regulations all require providers to document causal chains for substantial system modifications. The authors describe the EU framework as operationalizing this through mandatory risk management logs, incident notification with causal evidence, and renewed conformity assessments for any change altering a system’s risk profile. The community, they write, needs “new standards capable of documenting what changed, when it occurred, why it produced a specific outcome, and which interventions can systematically reverse it.” That sentence reads as a specification, and the rest of the paper is an attempt to meet it.
Why the problem resists a quick fix
The difficulty is not that the tools are missing. The paper maps six families of comparative explainers — feature attribution, probing, representation similarity, concept discovery, mechanistic interventions, and data attribution — and shows that each supports some desiderata naturally and others not at all. Feature attribution is fast and communicable but correlational and method-sensitive. Mechanistic interventions give the strongest causal evidence but are labor-intensive and architecture-dependent. No single family covers the checklist, which is why the authors insist that the desiderata govern how any instantiation is used and interpreted. The image that summarizes the problem is the transition report itself: a document that must link every claim to checkpoints, prompts, intervention settings, and controls, distinguishing what was merely observed from what was causally established. Building one is straightforward in principle, though the paper is explicit that such a report is a candidate reporting standard, not a compliance certificate. Building one that survives an auditor’s questions requires the reference model, the placebo, the seeds, and the disclosure — and the paper’s real contribution is showing that the discipline of producing it can turn a model update from an event nobody can reconstruct into evidence.
The full framework is presented in Comparing Explanations is Not Enough, Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models, a position paper by researchers at Scuola Normale Superiore, ISTI-CNR, and the University of Pisa.