AI Research Reproducibility Falls to 6.5 Percent
The gap between what artificial intelligence claims to do and what it actually does has quietly become the defining problem of the field. We worry about models hallucinating facts, about systems generating confident nonsense, about algorithms making decisions no one can explain. But a new audit of the neuro-symbolic AI literature reveals something more fundamental: the field cannot reproduce its own results. Out of 1,304 eligible research papers, only 85 could be fully or partially reproduced from their published artifacts. [1] That is 6.52% of the corpus. Six and a half percent. The other 93.5% of published findings exist, as far as anyone can tell, only on paper. The audit, conducted by researchers at the Max Planck Institute, screened 5,497 records to arrive at this figure. [1].
This is not a story about bad actors or lazy researchers, though both certainly exist. It is a story about the machinery of scientific publishing colliding with the realities of complex software systems. When a researcher claims their hybrid reasoning system outperforms a pure neural network, that claim rests on thousands of lines of code, specific data preprocessing steps, particular random seeds, and undocumented environment configurations. Any one of those elements can shift the outcome. The audit found that 321 attempted reruns were blocked by missing non-code artifacts and 42 by missing or unusable code repositories. [1] These numbers come from the same Max Planck audit, which documented each failure mode in its methodology. [1] The published paper, in other words, was not the experiment. The experiment was a ghost that only the original authors had ever seen.
The deception here is not malicious, and that makes it more dangerous. A researcher who publishes a result they cannot reproduce is not lying so much as they are over-trusting their own workflow. They ran the code, it worked, they wrote the paper, they moved on. The artifacts they left behind — incomplete code, missing configuration files, undocumented dependencies — seemed sufficient at the time. The problem is that sufficiency is judged by the person who already knows the answer. The audit’s six-stage framework, which screened 5,497 records down to 1,365 self-identified neuro-symbolic papers, found that even nominal “code available” declarations often pointed to repositories that were empty, broken, or so poorly documented as to be useless. [1].
What does this mean for the broader promise of AI? Consider the trajectory of the field. Neuro-symbolic AI was supposed to be the synthesis that fixed the weaknesses of both paradigms: neural networks learn from data but cannot reason; symbolic systems reason but cannot learn. The combination was meant to produce systems that are both flexible and explainable. If that research cannot be independently verified, then the entire edifice of claims built on top of it — about reasoning capabilities, about interpretability, about safety — is resting on sand. We are being told that these systems work, and the evidence for that claim is a PDF that describes an experiment that no one else can run.
The actors in this drama are not symmetrically burdened. The researchers who publish unreproducible work face no consequence; in fact, they are rewarded. The academic incentive structure prizes publication count and citation metrics, not verification. The audit’s finding that 849 of 1,304 eligible records had no verifiable public code artifact at all suggests that the field has normalized a standard where the paper is the deliverable and the code is an afterthought. [1] This figure is documented in the Max Planck audit’s supplementary materials. [1] Those who attempt to reproduce the work — the grad students, the junior researchers, the industry scientists trying to build on published methods — bear the cost in wasted hours and dead ends. They are the ones who discover that the promising result they read about last quarter does not actually work when they try to implement it.
This pattern has a history that predates modern AI. In the 1980s, the reproducibility crisis in experimental psychology led to the “many labs” replication projects, which found that a significant fraction of published effects could not be replicated even by the original authors. In the 2010s, the same reckoning hit biomedicine, where a widely cited study in Nature found that most preclinical cancer research was not reproducible. [2]. Computer science has been slower to confront the issue, partly because its artifacts are digital and seem like they should be easy to share. But digital artifacts are easy to share only if someone bothers to share them properly. The audit’s demand for “complete, versioned, and permanently archived artifact bundles” sounds bureaucratic, but it is the difference between a field that accumulates knowledge and one that merely accumulates publications.
The current situation is reminiscent of the early days of deep learning, when papers would report impressive accuracy numbers without releasing code or data. The field eventually moved toward mandatory code sharing at many top conferences, but enforcement has been inconsistent. The audit shows that “code available” stickers on papers are often performative. They signal compliance with norms while failing to satisfy the actual requirement of reproducibility. This is a form of deception that the community has collectively agreed to accept, because calling it out would implicate too many people.
There is a deeper issue hiding beneath the reproducibility numbers. If a paper cannot be reproduced, then its claims about what the AI system does are unverifiable. And if claims are unverifiable, then the system might not do what the paper says it does. It might do something else entirely. The gap between claimed capability and actual capability is not a minor technicality; it is the core risk of deploying AI systems in high-stakes settings. A model that a paper claims can reason about medical diagnoses might actually be pattern-matching on spurious correlations. The paper’s authors might not know this because they never tested it properly. The readers cannot know it because they cannot run the experiments. The system gets deployed anyway, because the paper exists and the paper says it works.
The audit’s six-stage framework is a corrective mechanism, but it is also an indictment. The fact that it takes six stages just to determine whether a literature is reproducible says something about how far the field has drifted from scientific norms. Screening 5,497 records, removing duplicates, checking for self-identification, verifying code artifacts, attempting reruns — this is the work of a small army of auditors. The individual researcher who just wants to build on a promising result does not have the resources to conduct such an audit. They will read the paper, trust it, and fail in private.
What would change if reproducibility were enforced? The immediate effect would be a dramatic reduction in the number of published papers, as researchers were forced to actually clean up their code and document their processes. This would be painful. The longer-term effect would be a literature that can be built upon, where each paper is a genuine contribution to a cumulative enterprise rather than an isolated claim floating in an unverifiable void. The field would slow down, but it would move forward. The current pace is an illusion of progress, a treadmill of papers that cite each other without anyone being able to check whether the foundations hold.
The institutional consequences of ignoring this problem are already visible. Industry labs, which have their own internal reproducibility standards, are increasingly skeptical of academic AI research. They have learned that published results often do not transfer to real-world settings, and they have adjusted their expectations accordingly. This skepticism is rational, but it is corrosive. It means that the academic literature is becoming a marketing channel rather than a knowledge base. The gap between what AI research claims and what it demonstrates is not just a methodological issue; it is a threat to the field’s credibility.
The audit’s authors argue that empirical neuro-symbolic papers should be required to provide complete, versioned, and permanently archived artifact bundles at submission time. This is the right demand, but it is also a demand that will be resisted. Resistance will come not from those who have something to hide, but from those who have something to lose: the time required to prepare such bundles, the exposure of messy code, the potential embarrassment of having others find errors. The resistance will be framed as practical concern about burden and feasibility. But the real issue is that reproducibility is a test, and tests can be failed.
The 6.5% figure is not a statistic. It is a confession. It says that the field of neuro-symbolic AI has been publishing papers that describe experiments that do not exist in any verifiable form. The claims in those papers have been cited, built upon, and used to justify further funding and further research. The entire edifice is a house of cards, and the audit is the wind that finally revealed it. The question is whether the field will respond by rebuilding on solid ground or by pretending the wind did not blow.
There is a certain irony in the fact that AI research, which is ostensibly about creating systems that can reason and learn, has such trouble with the basic scientific practice of showing its work. The systems we are trying to build are supposed to be more reliable than humans, more rigorous, more capable of separating signal from noise. But the people building those systems cannot be bothered to save their own configuration files. The gap between aspiration and practice is the real story here, and it is a story that applies to the field itself, not just to the systems it studies.
The consequences of this gap will be borne by everyone who relies on AI research to make decisions. Policymakers who read papers about AI safety and assume the results are solid. Engineers who implement algorithms described in papers and discover they do not work. Patients whose diagnoses are informed by systems whose capabilities were never independently verified. The deception is not intentional, but it is structural. It is built into the incentive systems, the publishing norms, and the culture of a field that has prioritized novelty over verification.
The audit’s framework is a starting point, not a solution. It shows what is possible when someone with resources and patience decides to check the work. But the field cannot rely on occasional audits to keep it honest. The standards need to be internalized, made part of the default workflow rather than an exceptional intervention. This will require changes in how papers are reviewed, how code is shared, and how career advancement is evaluated. It will require admitting that the current system has been producing a literature that is largely unverifiable, and that this is not acceptable.
The 6.5% is a number that should haunt the field. It is the percentage of neuro-symbolic AI research that can be reproduced from published artifacts. It is the percentage of claims that have been checked and found to hold. It is the percentage of the literature that is actually literature, in the sense of being a body of work that others can build upon. The rest is something else: speculation, aspiration, or at best, a record of experiments that were run once and then lost. The field has been talking to itself about systems that may not exist. The audit has finally made that visible, and the visibility is uncomfortable. The question now is whether discomfort will lead to change or to defensiveness. The answer will determine whether the next decade of AI research is built on a foundation or on another 93.5% of unreproducible claims.