A study now circulating in preprint form Automated research systems now propose ideas asks who gets to decide an idea is good — and what happens when that decider is another language model with no ground truth to answer to. [1] The answer, in this work, is that a panel of models can stand in for the human evaluator between a hypothesis and its acceptance, and that such a panel can be optimized against until it applauds claims that are mostly false.
A Testbed Where Truth Does Not Need a Referee
The setup is narrow, and that narrowness is the point. Researchers built a space of conditional interaction claims about two fully tabulated benchmarks: NAS-Bench-201, whose cell structure yields 1,740 directional claims, and HPOBench, whose tabular MLP benchmark yields 296 claims Automated research systems now propose ideas. A claim names an edit, a condition and a direction, and asserts that the edit’s effect is higher or lower when the condition holds than when it does not. Every claim can be checked exactly against the table. On NAS-Bench-201, 198 of the 1,740 claims are true, a random precision of.114; on HPOBench, 41 of 296 are true at a threshold of half a percentage point, a random precision of.139.
In ordinary ideation research, a person or an LLM judge decides whether a proposed hypothesis is worth pursuing or novel. Here the table decides whether it is true, and a calibrated panel of other language models decides whether it is surprising. Neither judgment requires a human in the loop. The authors are explicit that their surprise score measures excess truth relative to a calibrated LLM panel and nothing more — not surprise to experts, not surprise to the literature, which they did not measure. [1]
The Two-Stage Trick That Beat the Panel
The main positive result is simple, and it was found by an optimizer rather than designed by a person. It rests on one generator and one benchmark. A harness asks the generator to draft a pool of 30 candidate claims, then makes a separate call to keep 10 of them, under a rule the optimizer wrote: retain the claims the generator believes at 55 percent or more likely but expects other models to doubt at below 50 percent. For Sonnet on NAS-Bench-201, this step raised the calibrated excess-truth score by 1.77 per run over direct generation, by 1.95 over a fresh batch of direct runs, and by 1.96 over an ablation that keeps only likely-true claims. All three tests returned p below.001, though the authors are careful to note that the three share the same 24 harness runs and are one confirmation against three comparators, not three replications. Precision rose from.79 to.83.
The ablation distinguishes replaced judgment from improved prompting. Keeping the same strategy and the same pool of 30 but selecting the 10 most likely to be true erased the gain entirely: the score fell 1.96 below the confirmed harness and 0.19 below direct generation, at the same precision. Generating a large pool at the same cost does nothing; the selection rule carries the result. The rule needs no external feedback when it runs, which is precisely what makes it cheap to apply.
The Decider Is Not Doing What It Says
Here the study turns self-critical. The rule’s wording describes a process — belief against expected doubt — that the data does not confirm. Measured against Sonnet’s separately elicited blind prior, the kept claims are not the ones Sonnet rated at least 55 percent likely: 27 percent of kept claims against 45 percent of those not kept, and that blind prior does not discriminate truth within the pools at all, with an AUC of.51. Only 5 of 24 selection replies mention other models. The kept claims are, however, enriched for claims the leave-one-out panel rated below 50, at 67 percent against 54 percent. Whatever the generator does in that second call, its selections favor claims the panel doubts and are more often true — but the paper cannot show that it follows its own stated probabilities. The explanation remains a hypothesis.
The harness’s 240 hypotheses contain only 93 distinct propositions, and the single largest contribution comes from a false claim that direct generation proposes often and the panel expects — setting an edge to a 3x3 convolution helps less when the input edge is none — which the harness avoids. That claim holds in the stated direction on all three datasets but misses the threshold on CIFAR-10 by 0.07 percentage points. Drop the ten propositions that contribute most from both arms and the gain against the original direct runs falls to 0.07, against the ablation to -0.02, and against the fresh direct runs to 0.48. Resampling propositions jointly with runs widens the 90 percent interval from roughly 1.3 to 2.25 out to 0.38 to 3.30. The gain is real on fresh runs and, under the main panel, generalizes beyond the particular claims sampled — but it rests on a few recurring propositions, as one would expect from a 1,740-claim space a generator samples repeatedly.
Where the Machine’s Verdict Replaces the Expert’s
One harness was evolved against an LLM judge’s novelty-plus-quality score, and it raised that score on both benchmarks and for all three generators while cutting precision by 46 to 68 percent. The evolved strategy forbids two claim types “because experts find them obvious” — conditioning on the current value of the edited parameter, and conditions that make the edited component unused — and these are the claim types most often true. Before any optimization, a screen of 72 runs had shown what the judge rewards: its novelty rating correlates with the mean prior of a run’s hypotheses at -0.58, and its quality rating with the number of true hypotheses at +0.53.
The judge applied a style preference, the optimizer learned it, and precision collapsed. This is a case study of one harness from one optimizer run, not a general law about judges. Post hoc, the loss holds under every truth definition tested, for all three generators and all eleven definitions. On HPOBench every judge-optimized claim is of the “Other” type, and these hold far less often than direct generation’s claims of the same type,.35 to.43 against.80 to.96, with the 90 percent intervals of the difference all below -0.3. The type mix alone would predict precision of.80 to.96, so the harness’s claims are worse within type, not merely of a different type.
Removing each hypothesis’s one-sentence “why” still left the judge preferring the judge-optimized sets in four of six pairs, always through novelty, never quality. A retest with the original prompt reproduced the gain, but single-run totals barely correlate between judgings, from -0.22 to.62, so only arm means reproduce. A second judge, GPT, rated the judge-optimized sets higher in novelty and lower in quality in all six pairs. Both judges read the harness’s claims as more novel; precision, which needs no judge, shows what they cost.
The Measurement That Sounds Like Knowing
A second finding concerns a different kind of human skill: the ability to run 100 cheap experiments and read the results. GPT, given 100 cheap probes whose values it could read, did better than when given the same probes with shuffled or withheld values, on both benchmarks. The probes and the proposal call are identical across arms; only what the values say differs. So GPT’s proposals use what the probes say, not merely the act of designing experiments or a longer prompt.
A cheaper single-call baseline of 40 claims matched measurement on HPOBench at about two thirds of the tokens, 11.0k against 17.4k, and on NAS-Bench-201 measurement pushed precision below the pre-registered bound, from.69 to.50, so its gains there do not count as improvements under the study’s own reading rule. Measurement never helped Sonnet, which followed unrepresentative probes and lost precision; 37 percent of Sonnet’s claims were conflict claims, against 5 percent for Opus. A prompt follow-up suggests Sonnet discounts evidence described as noisy but not evidence described as unrepresentative. The human researcher who designs 100 cheap probes and knows which results to trust has not been replicated here. The model reads the values, but on one benchmark it reads them worse than simply asking for more claims.
The Prior That Flattered the Model
The study’s negative results show a measurement artifact manufacturing a finding. The original surprise metric asked the panel only for the probability that the edit’s effect is higher, and derived the lower-direction probability by complement. The panel’s mean probability of “higher” was 21 to 46, so no true lower claim was eligible in any of the four model-benchmark pairs, while 57 percent of true claims on NAS-Bench-201 and 54 percent on HPOBench are lower-direction. On HPOBench the two wordings of the same proposition received opposite labels for 41 to 60 of 60 propositions. Measurement shifted proposals toward higher claims, the only direction the metric credited, and this shift accounts for +1.02 of Opus’s +1.04 and +0.68 of Sonnet’s +0.88.
An external audit caught it. The numbers reproduced exactly from the caches, but the finding did not survive. Stage 12 re-elicited a three-way prior, and under the corrected metric the measurement effect for Claude generators is refuted in three pairs and inconclusive in the fourth. Keeping the original ratings for higher claims and using the new ones only for lower claims gives the same picture, so the reversal comes from the missing lower-direction prior, not from the new question format. The lesson the authors draw is that each failure they met had a cheap check that would have caught it — report precision on a checkable subset next to any judge or surprise score, and elicit priors over every outcome rather than one direction.
What the Calibrated Panel Cannot See
Both benchmarks are public, and the strongest priors predict truth with AUC.88 to.95, so the models may have seen the tables or papers about them. Contamination that the panel shares is discounted by construction, because the calibrated prior already predicts those claims. But a generator that recalls more than its panel earns credit for recall rather than reasoning, which is one reading of the panel dependence the study documents: on NAS-Bench-201, measurement raises the score for Opus and GPT under all eight panels that include the other frontier model, and under none of the six that do not. The authors read this as a hypothesis, not a finding, because no panel has an AUC between.74 and.87, so panel quality and the presence of the other frontier model are confounded.
The stage-3 harness fits the contamination reading less well: Sonnet’s own blind prior predicts truth less well than Opus’s, with an AUC of.71 against.88, the gain is positive when scored against Sonnet’s own prior at +1.25 per run, and the kept claims are not those that prior rates likely. The authors cannot rule contamination out. A private or post-cutoff table would settle it, but the one private-table transfer they ran tested only the judge-optimized harness.
The Data Point That Reverses the Frame
The study reports 17 pre-registered stages, over 4,500 model calls, and a table listing every pre-registered criterion and its verdict. [1] Of 19 harness evaluations on fresh NAS-Bench-201 runs, none beats direct generation on the corrected count of true, non-obvious claims after Holm correction. On the calibrated excess-truth score, two Sonnet harnesses do: the judge-optimized one, at precision.43, and the stage-3 one, at.83. Cross-domain retrieval, described as the largest single gain reported for a recent ideation system, did not help on any metric. The validity fitness, also grounded in truth, produced a harness that played safe.
The reversal is this: the one intervention that raised verified surprise without losing precision was found by an optimizer searching against a score, confirmed in fresh runs, and then shown to rest on a few recurring propositions and to fall short of significance against the generator’s own prior once propositions are resampled. The one intervention that reliably raised the judge’s ratings was evolved against that judge and collapsed precision for every generator. The human role under pressure is not the hypothesizer. It is the evaluator, and the study’s evidence is that when a model takes that seat and is then optimized against, it rewards novelty of style over truth, and the ideas it selects are true less often than the ones a table would have kept. The paper’s own summary of what survives is narrower than its title: a selection step that favors claims the panel doubts and that hold, tested with one generator on one benchmark, and a set of cheap checks that would have caught every failure the authors met along the way. The audit that caught the direction bias was external to the analysis, and it is the strongest argument in the paper for keeping a human in the loop.