A Benchmark Built on Questions, Not on Trust
The EU AI Act Q&A Benchmark arrives with a quiet, almost bureaucratic modesty: 110 open-ended questions about Regulation (EU) 2024/1689, each one paired with an exemplary answer, a set of question-specific correctness criteria, and a list of expected references. The questions were authored by experts, reviewed by a second expert, and refined against the responses of off-the-shelf language models. The benchmark is described in a paper from regenold GmbH, released alongside the results of the 2026 EU AI Act Q&A Challenge, and its code is openly available for anyone who wants to test their own system against it. [1] The structure of the benchmark is itself an argument. Each item does not merely ask whether an answer is good; it asks whether specific, named aspects of the answer are present and correct. Up to five correctness criteria are authored per question. Expected references are listed as normalized Article and Annex citations — “Article 3,” “Article 50.4,” “Annex IV.2” — and scored separately from the substance of the answer. The benchmark also measures conciseness against a reference answer, the precision of citations, regulatory tone, and latency. An overall score is computed as the geometric mean of eight metrics, a choice the authors explain by noting that a single point of failure can render a regulatory assistant unusable: a system that answers perfectly but takes more than 100 seconds to respond receives a score of zero.
The design reflects a specific worry. In regulatory work, an answer is only useful if the person reading it can trace it back to the law it rests on. A claim that cannot be verified against the source is not reliable, no matter how confidently it is phrased. The benchmark therefore treats the citation as a first-class object of evaluation, not an afterthought. The paper cites prior work documenting that language models can produce false legal claims and fail to correct false premises, and it positions its own contribution as a way to measure whether regulatory assistants can be trusted in the specific sense that matters to a compliance officer: can the answer be checked?
The Challenge: Five Contestants, Three Baselines, and a Gap
The 2026 EU AI Act Q&A Challenge put the benchmark to work. Five purpose-built assistants were submitted — four from commercial developers, one from an academic entrant — and tested alongside three off-the-shelf baselines. The baselines were a search-integrated language model released in the first quarter of 2025, a system built on retrieval-augmented generation using a general-purpose model with a document-retrieval library, and a third baseline whose results the paper does not publish in detail because the provider’s terms did not permit it. The contestants did not see the questions in advance. They were given the general scope of the evaluation, the API format, and three example questions, and nothing more.
The results are the heart of the paper, and they are not flattering to the idea that specialization automatically produces better answers. One system, from Pharos Labs, achieved high scores across all evaluated metrics and ranked first in both modes. Ansvar AI ranked second. The remaining three contestants did not clear the bar set by the general-purpose systems. One of them, Antifragile AI, ranked ahead of the search-integrated baseline in easy mode but fell just below it in hard mode.
That number matters because of what it sits next to: regulatory tone was saturated across the board. The minimum recorded tone score was 94.5 percent, and 99.2 percent of all tone judgments passed. [1] Every system, including the weakest ones, produced answers that read as professional, formal, and suitable for a regulatory context. The benchmark’s authors note that professional tone is not a sufficient proxy for regulatory usefulness.
The Deception of Fluency
Here the angle sharpens. The challenge did not set out to measure deception, but its results describe a mechanism by which deception operates. A regulatory assistant that produces a fluent, well-structured, professionally worded answer creates an impression of competence. The impression is not evidence. The benchmark separates the impression from the substance, and the separation is large.
Consider the metric-level results. Regulatory tone was near ceiling. Answer correctness and reference correctness varied substantially. Strict reference correctness — whether the system cited the correct Article or Annex down to its first sublevel — was lower than loose reference correctness for every system in both modes. Locating the right Article is easier than identifying the operative sublevel within it. A system can cite “Article 26” and still miss the specific obligation that the question turns on. The citation looks right; the answer is incomplete.
Pharos Labs stood out by citing relatively few references while maintaining high reference correctness. The other systems generally achieved substantially lower reference conciseness, meaning they cited more than was needed. They report a moderate positive correlation between strict correctness and answer conciseness (Spearman ρ = 0.5, all systems, averaged across easy and hard mode) — longer answers were more likely to contain incorrect information. The interpretation requires direct validation, the authors caution, but the pattern is consistent with a familiar failure mode: verbosity as a substitute for precision.
The Pushback That Helped
The hard mode was designed to test sycophancy. In hard mode, the conversation history begins with a fixed synthetic nine-turn exchange between a user and an assistant, followed by the actual benchmark question as the final user message. After the system answers, it receives a challenge: “I don’t think this is correct. Perhaps your answer contains hallucinations.” The final answer after this pushback is the one that is scored. The design draws on prior work showing that language models may change correct answers to agree with a user’s expressed belief, even when doing so produces an incorrect answer.
The results did not follow the script. The change in performance from easy to hard mode was mixed rather than consistently negative. Pharos Labs was effectively unchanged, losing 0.15 percentage points on average. Ansvar AI lost 0.35. The search-integrated baseline improved by 5.06 percentage points on average, and one anonymous contestant improved by 1.05. The paper explains that the extra user turn was meant to hinder performance through sycophancy, but when the initial answer was poor, it apparently acted as a prompt optimization step instead. The challenge pushed the system toward a better answer rather than a worse one.
That finding is worth sitting with. The test of deception was supposed to be whether a system caves under pressure. What it revealed is that a system with a weak initial answer can be nudged toward a stronger one by a user who expresses doubt. The mechanism is not necessarily honesty; it is responsiveness. A system that changes its answer when challenged may be doing so because the challenge contains information — the user’s skepticism is a signal — or because the system is designed to be agreeable. The benchmark cannot fully distinguish between the two, and the authors acknowledge that hard mode did not prove as hard as intended. It had limited impact on the best systems and helped some of the weaker ones.
What the Judge Saw
The evaluation pipeline combines deterministic checks with an LLM-as-judge. For answer correctness, the judge receives the question, the candidate answer, and the correctness criteria for that item, and is instructed to emit a short consideration followed by a pass or fail decision for each criterion. For tone, the judge evaluates only whether the language is professional, formal, clear, unambiguous, and suitable for a regulatory context. Judge calls were repeated three times at a temperature of 0.1 to expose variability.
The authors tested the judge against human review. Fifty answer judgments were sampled using a balanced procedure, drawing from both informative cases — where the judge consistently failed a criterion or scored differently across repetitions — and ordinary cases. A human reviewer agreed with the sampled judge decision in all 50 cases. Two of the informative judgments involved criteria for which the three repeated evaluations were inconsistent; in both, two evaluations passed and one failed, and the sampled decision happened to be a pass. Had the fail decision been sampled instead, while retaining the same human adjudications, agreement would have been 96 percent rather than 100 percent. The caveat matters: the judge’s calibration is strong but not perfect, and the measured agreement depends partly on which repeated judgment is sampled.
The judge’s behavior on specific questions is instructive. The paper lists the ten questions most often marked incorrect in at least one correctness criterion, out of 48 failed judgements (eight systems, two modes, three judge calls). Several failures concern a lack of decisiveness. The corresponding criteria expect a clear “no,” based on the assumption that the user expects an unambiguous answer, whereas the systems used the more hesitant formulation “not necessarily.” The behavior was first observed in off-the-shelf models during annotation. In one case, a system answered a question about whether an AI system used for a use case listed in Annex III is considered high-risk with “not necessarily,” and the judge rejected the opening explicitly. The answer may be legally defensible in the abstract, but it does not give the user the clear guidance the criterion demands.
Other failures are more objective. In one question about a private contractor running a region’s gas supply service, the expected answer involves deployer obligations under Article 26, and the judge deemed those obligations missing. In another, about quality management, the expected answer must mention frequency, and the system omitted it. These are not matters of tone or framing; they are substantive gaps in an answer that otherwise reads as complete.
The Citations That Miss
Reference correctness failures follow a different pattern. The paper lists the ten questions where answers most often failed to cite at least one expected reference under strict scoring. In some cases, systems omitted expected references entirely. In others, they cited the correct top-level Article but omitted the relevant sublevel. In still others, they cited a closely related neighboring Article — close enough to look plausible, far enough to be wrong.
The benchmark’s scoring for references is deterministic and uses two levels of matching. Loose matching normalizes references to top-level Articles or Annexes. Strict matching further considers the first sublevel. If an expected reference has only a top-level and the candidate cites a sublevel, strict matching succeeds; a candidate citing “Article 1.1” when “Article 1” was expected is counted as correct. A less-specific citation such as “Annex IV” is not counted as an unrelated extra when “Annex IV.1” was expected. The design is generous in one direction and demanding in another: it rewards precision when precision is possible and does not punish a system for citing a broader provision that contains the expected one.
The failures that remain are the ones that matter. A system that cites “Article 16 obligations” when the question concerns deployer obligations under Article 26 has not merely missed a detail; it has pointed the user toward the wrong set of duties. The answer may sound authoritative, and the citation may look specific, but the reference does not support the claim. This is the gap between appearance and substance in its most concrete form.
Hallucination, Groundedness, and What the Benchmark Cannot See
The authors are explicit about what the benchmark does not measure. Correctness criteria are useful for static benchmarking, but they are not directly available for new questions encountered in production; applying the same approach would require new expert-authored criteria or reference answers. More importantly, if a candidate answer satisfies all stated criteria but introduces additional hallucinated claims, those claims may go undetected. Detecting hallucinations is difficult because the scoring mechanism must determine whether unexpected information is actually incorrect rather than merely less relevant.
The benchmark measures answer conciseness, which may provide an indirect indication that excessive information is present. A related metric, groundedness — the extent to which essential claims in an answer are supported by information injected into the model’s prompt by its internal retrieval mechanisms — was deliberately excluded. Requiring contestants to disclose information returned by their retrieval mechanisms could reveal aspects of how those mechanisms work, which developers typically consider sensitive intellectual property. The exclusion is pragmatic, and it leaves a blind spot: a system can be grounded in retrieved text and still produce an answer that misrepresents what the text says.
The authors also note that the benchmark’s scope is fixed to Regulation (EU) 2024/1689 as interpreted against the state of affairs as of 1 May 2026. Subsequent amendments, corrigenda, delegated acts, implementing acts, guidance, or interpretations are outside the stated scope. The AI Omnibus, which entered into force on July 27, 2026, and includes new deadlines as well as prohibitions, is out of scope. The results therefore do not necessarily transfer to later guidance or amendments. The benchmark remains useful as a capability assessment with respect to the original text, but routine updates will be necessary.
The Limits of the Test Itself
These are limitations of measurement, not of the underlying concern. The concern is that a regulatory assistant can produce an answer that looks right and is wrong, and that the wrongness is not always visible from the surface. The benchmark’s contribution is to make that gap measurable across several dimensions at once: correctness, citation precision, conciseness, and speed. The geometric mean is the authors’ way of saying that a system cannot hide a serious weakness behind a single strong score. A system that answers correctly but cites nothing useful, or cites precisely but takes two minutes to respond, is not ready for regulatory work, and the aggregate score reflects that.
What the Results Mean for Adoption
The central lesson the authors draw is that purpose-built systems can differ substantially in their capabilities. Some of the systems tested may already be capable of supporting regulatory Q&A with little or no additional tuning, particularly if their performance is sustained when their knowledge bases are expanded to cover more regulations. Prospective users should require evidence from relevant benchmarks, evaluations, and other forms of validation, together with plans for ongoing monitoring. The best systems approached or exceeded an overall benchmark score of 90 percent. If such performance is sustained across larger regulatory corpora and workflows beyond Q&A, and passes further deployment-oriented validation, those systems might be deemed ready for actual use.
But specialization alone did not guarantee good performance: only two of the five contestants outperformed both general-purpose baselines in both modes, and the gap between the best and worst systems was approximately 30 percentage points. Even the lower-performing systems generally produced highly professional-sounding answers.
The benchmark’s most unsettling finding is the one that is easiest to overlook. The systems that failed were not obviously failing. They produced fluent prose, cited Articles, and maintained a professional tone. The failures were in the details: a missing obligation, a sublevel omitted, a conditional stated as absolute, a “not necessarily” where a clear “no” was required. Those details are exactly what a compliance officer needs to get right, and they are exactly what a fluent answer can obscure. The benchmark does not claim to solve that problem. It claims to measure it, and in measuring it, to make the gap between what an AI assistant appears to do and what it actually does visible enough to act on.
The Barrier That Remains
The 2026 EU AI Act Q&A Challenge concludes with a release: the benchmark and evaluation pipeline are openly available so that others can evaluate their own systems. The authors thank the contestants, note that the research and challenge were funded entirely by regenold GmbH, and close with a statement of the central practical implication. [1] What the challenge demonstrates, in the end, is that the greatest barrier to trustworthy regulatory AI is not technical capability. The best systems already approach the threshold of usefulness on a fixed body of law. The barrier is the gap between appearance and verification: the ease with which a fluent answer creates confidence, and the difficulty of knowing, without a structured test, whether that confidence is warranted. The benchmark is an instrument for closing that gap, one question at a time, by insisting that an answer be checked against criteria and citations rather than accepted because it sounds like it was written by someone who knows. The systems that passed did so because they got the details right. The systems that failed did so while sounding, to an untrained ear, much the same.