The Question Everyone Asks Is the Wrong One
The obvious question about large language models in data work is whether they beat the old statistical tools. The 2026 study “When Do LLMs Help with Feature Selection? A Systematic Empirical Study” answers it, and the answer is mostly no — classical selectors such as mRMR, LASSO, Boruta, and XGBoost importance generally predict better, even when the language models are handed the same feature budget under strictly train-only fitting. [1] But that verdict, however clean, is not where the interesting deception lives. The interesting deception lives in the second experiment, the one where the researchers deliberately lied to the model and watched it believe every word.
What the Benchmark Actually Measured
The setup is disciplined. Eighteen datasets, split into ten for reinforcement training, two for checkpoint validation, and six held out entirely from training, few-shot examples, and checkpoint selection. Ten selectors compete: a no-selection control, four classical methods, frozen zero-shot and few-shot Qwen2.5-7B, a frozen Llama-3.3-70B, and two Qwen models adapted with Group Relative Policy Optimization. Every one of them picks a subset of columns, and a fixed XGBoost model then judges the subset by ROC-AUC for classification or R-squared for regression. Five seeded partitions, task-separated metrics, dataset-wise ranks. The design is built to stop a selector from flattering itself.
The Scoreboard and Its Small Print
Classical methods lead. No Selection reaches a mean classification AUC of 0.909 and a mean rank of 2.67; LASSO hits 0.910 at 67 percent retention; XGB importance sits at 0.897. Zero-shot Qwen-7B lands at 0.819 AUC and 0.571 R-squared with a mean rank of 7.83. The GRPO-tuned v1 improves regression over its frozen base (0.596 against 0.571) but drops classification to 0.799, and the second tuning run, v2, falls to a rank of 9.17. The paper is careful about power: with five partitions, even a unanimous 5-0 paired difference yields a minimum two-sided exact Wilcoxon p of 0.0625 before correction, so all comparisons are read descriptively. The authors say as much, and the honesty is part of the story.
The Control That Removes the Excuse
A skeptic could object that classical selectors saw the full dataset while the language models only saw summary statistics. So the authors built a second experiment: mRMR, XGB importance, and LASSO fitted strictly on the 60 percent training split, truncated to a budget matching the RL model’s 33.4 percent retention. The gap barely moves. XGB Top-k reaches 0.890 classification AUC against 0.799 for the tuned model; mRMR Top-k reaches 0.663 R-squared against 0.596. On five of six datasets the classical control wins across all five partitions. The one exception, parkinsons_telemonitoring, is where the tuned model edges ahead — and it is also, not coincidentally, where the next experiment finds its most dramatic effect.
Now Lie to the Model
Here is the heart of it. The researchers took zero-shot Qwen-7B and ran it three ways on the same six held-out datasets. In the Original condition the model saw real feature names — age, cholesterol, whatever the column actually meant. In the Anonymized condition those names were replaced with opaque identifiers. In the Shuffled condition the real names were kept but reassigned to the wrong columns, a strict derangement with zero fixed points, then mapped back to their true positions before evaluation. Dataset and target names were masked throughout. Crucially, every condition used the exact same number of features per completion, so subset size could not explain any difference. The model never saw a single row, only column names and marginal statistics.
What the Model Claims to Know
The results separate two kinds of deception, and the first is the one we want from these systems: real names carry real information. On regression, Original names beat Anonymized by 0.1964 mean R-squared and beat Shuffled by 0.1606. On parkinsons_telemonitoring, R-squared jumps from 0.296 to 0.712 — the model extracts genuine domain signal from the feature names without ever touching the data. On classification the gain is far smaller, 0.0159 and 0.0084 AUC, and dataset-dependent. So semantic priors help, but they help unevenly, and mostly where the task rewards knowing what a variable means.
The Second Deception Is Ours to Notice
Then the Shuffled condition turns the microscope around. When plausible names are attached to the wrong columns, performance collapses. On ames_housing, R-squared falls from 0.886 to 0.542 — a drop of 0.344 — because the model confidently selects features whose labels describe something else entirely. The paper’s most damning number, on ames_housing: Original and Shuffled selections share a prompt-label Jaccard of 0.363 but a true-column Jaccard of 0.000. [1] In plain terms, the model picks the same words it would have picked anyway, and those words now point at the wrong data. It is not reasoning about the table, or so the pattern suggests. It appears to be reading the label and trusting it. Strip the label of its anchor and, by this reading, the model has no way to tell.
The Feedback Loop Nobody Instrumented
This is the gap the benchmark exposes, and it is structural rather than incidental. The model’s confidence comes from the semantic prior, not from the target. When the prior is accurate, the selection looks smart; when the prior is misapplied, the selection looks just as smart and is worse. The researchers controlled cardinality precisely because changing feature names also changes how many features a model picks, confounding semantics with subset size — a confound that would have hidden the effect in either direction. The controlled design is what lets them say, with evidence, that the model is responding to meaning rather than to statistics. And meaning, unlike a correlation coefficient, cannot check itself against the outcome.
Why the Tuned Model Does Not Fix It
Reinforcement learning was supposed to close this gap by rewarding downstream performance. It did not, at least not reliably. GRPO v1, selected on two validation datasets by mean reward of 0.8282 at checkpoint 100, produces a task-dependent trade-off: regression improves, classification degrades. GRPO v2, trained longer with fresh adapters and no validation-based checkpoint selection, raises classification AUC to 0.817 but worsens regression to 0.567 and rank to 9.17. The paper’s verdict is blunt — the adaptation changes selection behavior but yields no consistent predictive superiority over frozen or classical alternatives. [1]. [1] Optimizing a reward does not teach a model to distrust a label; it teaches it to produce subsets that score well on the datasets it was trained on.
The Sparsity Illusion
There is a quieter mismatch worth naming. The 7B selectors retain 29.7 to 33.4 percent of features — compact, tidy subsets that look like disciplined engineering. [1]. Llama-70B retains 60.8 percent. Classical methods range from 38.0 to 66.6 percent. If you judged by parsimony alone, the small models would win. But the Pareto analysis in the paper shows a trade-off, not a dominant strategy: the frontier includes Zero-shot Qwen-7B, Few-shot Qwen-7B, Boruta, XGB importance, LASSO, and No Selection. Compactness is not evidence of understanding. It is evidence of a budget the model was prompted to respect.
The Contradiction That Survives Improvement
So what does the study actually establish about where these systems help, deceive, and fall short of classical methods? Three things, and they sit uneasily together. First, semantic information is real: at fixed cardinality, authentic names improve regression substantially. Second, semantic information is fragile: misattributed names redirect selection toward poorer subsets with no internal alarm. Third, target-supervised statistics remain the stronger signal: classical selectors win at matched budget under strict train-only fitting, and no RL configuration tested reverses that. The technology can improve — the paper’s own limitations section points to higher-dimensional data, richer metadata, other foundation models, and alternative RL strategies as open ground — and the contradiction will persist, because the model’s advantage and its blind spot are the same mechanism. It knows what words mean. It does not know whether the words are attached to the right columns. Neither, unless someone checks, do we.
What to Do With a Model That Sounds Certain
The practical reading is not “avoid LLMs for feature selection.” It is that a selector built on semantics needs a target-supervised audit before anyone trusts it, and the audit has to be cheap enough to run every time the schema changes. The paper’s own framing — semantics as complementary information, not a substitute for statistical evidence — is the defensible position. The matched-budget control is the template: same budget, train-only fitting, compare against mRMR or XGB importance, and treat any win by the language model as a hypothesis rather than a result. The model will keep sounding confident. The Shuffled condition shows exactly how little that confidence costs it.
The Column That Was Never Checked
The deeper point is about what these systems are asked to be. A classical selector is transparent about its evidence: it reports a correlation, a coefficient, a gain, and you can inspect the number. A semantic selector reports a selection, and the evidence behind it is the meaning of a word — which is exactly as reliable as the person who named the column and exactly as invisible to the model as the data underneath. The benchmark’s contribution is to make that invisible dependency measurable: 0.363 label overlap, 0.000 column overlap, a 0.344 R-squared collapse on ames_housing. When the names are right, the model looks like it understands the domain. When they are wrong, it looks identical. That is the deception, and it is not the model’s alone.
The Question Worth Asking Instead
The better question, then, is not whether LLMs beat classical selectors — the 2026 evidence says they generally do not, at matched budget, with train-only fitting. The better question is what a selector’s confidence is actually tracking. For mRMR and LASSO, confidence tracks the target. For a language model reading column names, confidence tracks the semantics of the label, and the label can be shuffled without the model noticing. The paper’s 18-dataset framework, its cardinality-matched three-condition diagnostic, and its strict train-only control are a method for asking that question and getting an answer. The answer, in this study, is that the model’s semantic prior is real, useful, and unverified — and that the difference between those three adjectives is where every downstream failure begins. When Do LLMs Help with Feature Selection? A Systematic Empirical Study