A chest X-ray classifier is trained to spot pneumonia. Then a small share of its training images is quietly altered — a trigger patch added — and something measurable happens: the model gets better at the corrupted task and worse at the real one. That is the finding at the heart of a 2025 study on gradual data poisoning, and it is the clearest way to see what data poisoning actually does to a model rather than what the phrase merely implies. [1]
What Data Poisoning Changes, Stage by Stage
Data poisoning manipulates training, fine-tuning, evaluation, or retrieval data so that an AI system learns corrupted behavior or produces attacker-chosen outcomes. The definition carries three practical commitments. There is an identifiable input. There is a transformation or decision characteristic of the technique. And there is an outcome that can be evaluated against a stated objective. Strip away any one of those and the label describes an aspiration rather than a mechanism.
The source maps the information flow in five stages, and the map is worth walking forward and backward. First, an attacker gains influence over a data source. Second, mislabeled, biased, or trigger-bearing examples are inserted. Third, those examples survive filtering and deduplication. Fourth, the learned model or the retrieved context is altered. Fifth, the corruption activates — either as broad degradation or as a targeted backdoor. Each handoff has an owner, an input, an output, and a test, and that trace is where teams can catch a tiny targeted subset before it reaches a consequential output. The map is a causal aid, not a claim that every implementation uses five components; some systems combine stages and others repeat them in a loop.
The distinction that matters most is operational, not terminological. Data poisoning is routinely confused with an evasion attack applied only after the model is deployed. The two may share a visible feature, but they change the causal story. Different evidence would establish success, different resources would dominate cost, and different controls would prevent harm.
The Chest X-Ray Experiment: Measuring Corruption as It Grows
The radiology study built a generic ResNet pneumonia classification model and subjected it to BadNets adversarial attacks, a method that teaches a network to associate a trigger pattern with a chosen label. The researchers assembled poisoned datasets at three compositions — 2 percent, 16.7 percent, and 100 percent poisoned data — and evaluated each model on two test sets: a normal set and one containing poisoned images. [1] SHapley Additive exPlanations, or SHAP, provided a visual record of where the models were looking as corruption progressed.
The results moved in opposite directions at once. As corruption increased, interval analysis showed performance on the valid test set falling while the model learned to predict better on the poisoned test set. SHAP visualization showed the model’s attention settling on the trigger. Decreasing performance correlated with increasing focus on that patch.
The middle condition is the most instructive. In the 16.7 percent poisoned model, SHAP focus did not fixate on the trigger when the model saw the normal test set. [1] The corruption could potentially be masked unless the model was subjected specifically to poisoned data. At 2 percent, the effects were minimal. The authors noted that a minimum threshold for corruption may exist — a level below which the poisoning leaves little trace.
That threshold may be why the technique is dangerous in a way that ordinary bugs are not. A degraded model announces itself through error rates. A poisoned model can keep its headline numbers while quietly reweighting what it attends to, and the study’s SHAP images offer a visual record of that reweighting that a single accuracy figure does not. The work identifies areas of potential intervention for safeguarding models against adversarial attacks, and the visual evidence is the intervention’s starting point: a shift you have never seen rendered is a shift you are unlikely to monitor for.
The Same Mechanism in Genomes, Where Nobody Can Read the Labels
A 2025 preprint extended the question to genomic foundation models, and its findings sharpen the case for treating poisoning as a data-provenance problem rather than a model problem. Genomic models train on massive, publicly sourced corpora comprising trillions of nucleotide tokens. Unlike natural language, DNA sequences lack the semantic transparency that might allow model makers to filter out corrupted entries, which makes genomic training corpora particularly susceptible to undetected manipulation.
The researchers demonstrated two attack vectors. At the pre-training stage, using the Evo 2 and GENERator architectures, less than 1 percent of the training corpus — adversarially crafted sequences — selectively degraded generative behavior on targeted genomic contexts while leaving performance on unrelated sequences intact. They tested three scenarios: corruption of the TATA-box promoter motif, disruption of CTCF binding sites, and insertion of a short DNA sequence absent from all genomes in the training corpus.
At the fine-tuning stage, two further attacks worked. Poisoning a fraction of CTCF sites within a ClinVar-derived fine-tuning corpus installed a conditional backdoor in a LoRA-adapted model that activates almost exclusively in the presence of the trigger. Using embeddings extracted from the frozen Evo 2 7B model, targeted label corruption of downstream training data selectively compromised a clinically relevant variant classification task, with BRCA1 variant effect prediction as the case study.
The pattern repeats across both domains. A sliver of the corpus, less than 1 percent in the genomic pre-training case, buys a targeted effect. The rest of the model’s behavior stays intact, which is precisely what makes the corruption hard to notice. The genomic authors underscored the need directly: data provenance tracking, integrity verification, and adversarial robustness evaluation belong as integral components of the development pipeline.
Why the Gain Is Real and Why It Is Fragile
The honest case for studying data poisoning is that it produces a concrete, measurable gain — for the attacker. The chest X-ray model learned to predict better on poisoned test data. The genomic models learned to respond to a trigger. These are real improvements in a defined objective, and that is exactly why the defensive lesson is not abstract. A technique that reliably shifts model behavior with less than 1 percent of the corpus tells defenders where to look and what to measure.
The same logic defines the limits. A tiny targeted subset can be hard to detect in web-scale data, and that difficulty should shape data collection, architecture, permissions, evaluation, release gates, and monitoring from the beginning rather than appearing as an afterthought once development is complete. The chest X-ray study’s possible threshold and the genomic study’s sub-one-percent result both point at the same design question: at what point does a filtering or provenance check catch the corruption before it becomes learned behavior? Neither study answers it, and that gap is the honest state of the evidence?
Answering that requires evidence across the whole lifecycle, not just at the model layer. A control is meaningful only when its owner, scope, trigger, expected behavior, and verification method are explicit. Performance can be determined by surrounding data, interfaces, permissions, and people even when the underlying model is unchanged, so a useful account separates what the model learned from the product that decides when, where, and with what authority that learning is used.
Where Regulation Lags the Mechanism
The regulatory picture is behind the technical one, and the gap has a specific shape. Frameworks such as the NIST AI Risk Management Framework and the C2PA specifications address provenance and risk management in general terms, but neither tells an operator how to detect a 2 percent poisoned subset in a radiology corpus or a sub-one-percent trigger set in a genomic one. Those are empirical findings about a specific architecture, and they do not transfer automatically to a different model, dataset, or jurisdiction.
That is why the comparison between products and papers has to identify the unit of analysis. A study may isolate a model or algorithm, while a deployed service adds retrieval, routing, caching, policy, identity, user interfaces, and monitoring. Two systems can use the same headline term while implementing different parts of that stack. The question to ask is which component performs the defining transformation and which other components are necessary for the reported outcome.
The genomic preprint’s recommendation — provenance tracking, integrity verification, adversarial robustness evaluation — is a governance proposal as much as an engineering one. It asks model developers to treat the training corpus as evidence with a chain of custody rather than as raw material. The chest X-ray study’s SHAP visualizations ask evaluators to look inside the model’s attention, not only at its scores. Neither recommendation has a standard compliance test behind it yet.
The Next Step the Research Landscape Points To
What follows from these two studies is a shift in what counts as evaluation. The relevant measure is not whether a poisoning technique can produce one impressive result. It is whether defenses hold across representative conditions — distributions, failure categories, tail behavior, affected subgroups — rather than compressing everything into one average. The chest X-ray work showed that a model’s behavior on a normal test set and a poisoned test set can diverge sharply while a single accuracy number stays plausible. The genomic work showed that targeted degradation can coexist with intact performance on unrelated sequences.
Practically, that means precommitting to the finding that would falsify a claim. If no result could reverse an adoption decision, the evaluation is marketing. A preserved confirmation set, an untouched test set for controlled comparisons, and staged validation — shadow mode, canaries, rate limits, approval gates — turn a demonstration into evidence. Versioning the inputs needed to reproduce a result, from source data through preprocessing, model weights, configuration, retrieval index, and evaluation set, is what lets another team judge whether a claimed gain survives a different environment.
The research landscape also points toward earlier intervention: both studies are arguments for acting before an expensive or irreversible consequence. Identify the earliest observable precursor to the failure, set a threshold or rule, assign an accountable owner, and test recovery — whether that recovery means abstaining, falling back to a simpler system, requesting more evidence, escalating to a person, or rolling back a model.