The Answer Doesn’t Change, the Position Does
Laya is a language model built to answer multiple-choice questions. It reads a question once, scores a hidden state at a single [MASK] marker placed beside each option, and picks a winner. The architecture is compact: 421 million parameters, a two-layer option head with no positional tensor, and a documented parallel layout that the released checkpoint never actually uses. The design implies that where an option sits in the list should not matter, because the scoring mechanism has no component that reads position. A new re-engineering study of the checkpoint tests that implication directly, and the answer is that the implication fails.
Rotating the Options, Watching the Answers Move
The researchers took 951 questions verified by Qwen and presented each one under every cyclic rotation of its answer options — for questions with twelve options, six random permutations instead. [1] The released checkpoint changed its answer on 25.8 % of questions, with a confidence interval of 22.9 % to 28.7 %. [1] On the 605 English-language items alone, the flip rate was 19.7 %. Nothing about the questions changed between presentations. Only the order of the candidate answers did. The model, which appears to evaluate each option on its merits, was in fact reading something about the arrangement.
The Gold Answer Looks Better in First Place
Accuracy itself tracks the listing order. The study reports that the model’s accuracy varies systematically with where the correct option appears in the list, with a measurable advantage for options placed earlier. [1] This is not noise scattered evenly across the test set; it is a systematic advantage conferred by position. A user who receives Laya’s answer has no way to know whether the model identified the right option or simply favored the slot it occupied. The output looks identical either way — an answer delivered with the same flat confidence.
Flips Cluster Where the Model Is Least Sure
The instability is not uniform. Across confidence terciles, the flip rate was 54 % in the lowest tercile and 8 % in the highest. The model’s own certainty, in other words, partially tracks its own fragility: when it is unsure, it is also more likely to be swayed by where an option appears. [1]. That relationship is the most useful thing in the study for anyone deploying such a system, because it suggests that self-reported confidence carries real signal about order sensitivity even as it fails to flag the mechanism. [1]. A confidence score that drops when the answer is about to flip is a warning light, not a fix.
Training on Shuffled Rows Helps, but Only So Much
Fine-tuning with Laya’s own trainer on 2,818 teacher-verified rows and shuffling the options during training cut the flip rate to 7.2 %. The identical rows without shuffling produced 11.4 %, which means shuffling accounts for 4.2 points of the improvement. The remaining gap between 7.2 % and zero is the part that matters: even after explicit training against order dependence, roughly one question in fourteen still changes its answer when the options are rearranged. The remedy reduces the deception; it does not eliminate it.
Voting Over Orderings Buys Almost Nothing
A confidence-weighted vote across multiple orderings reached 69.7 % accuracy, against 69.1 % for a single base pass — at 4.1 times the compute. Four times the work, six-tenths of a point. The ensemble does not repair the underlying problem so much as average over it, and the averaging is expensive enough that the trade is hard to justify on accuracy grounds alone. What the vote does provide is a diagnostic: disagreement across orderings exposes the questions where the model’s answer is contingent on presentation, which is information a single pass cannot supply.
The Parallel Layout Closes the Gap
When the researchers fine-tuned using the authors’ parallel layout on the same rows, the flip rate fell to 0.5 %. That is the sharpest result in the study, and also the most ambiguous. The architecture that was documented but left unused in the released checkpoint may be the component that actually enforces order independence. If so, the published model was shipped in a configuration that its own documentation describes as unused — a gap between what the documentation describes and what the weights do.
Labels Made by Models, Tests Filtered by Teachers
Two caveats bound every number above. The labels are LLM-made, and the test sets are teacher-filtered, which means the ground truth against which flips are measured was itself produced by a model and then screened by another. A flip counts as an error only relative to a label that a language model assigned. This does not invalidate the finding — the rotation experiment measures self-consistency, which needs no external truth — but it does mean the accuracy figures describe agreement with a model-generated standard, not with human judgment.
The Gap Between the Spec and the Artifact
The deeper deception here is not that Laya answers differently when options move. It is that the model’s design documentation describes a system whose scoring has no positional component, and the shipped checkpoint behaves as though it does. Anyone reading the architecture would conclude that order cannot matter. Anyone running the rotation test learns otherwise. The 25.8 % flip rate is the size of that gap made visible, and the 0.5 % result from the parallel layout is evidence that the gap was closable all along.
What the User Never Sees
A person asking Laya a question receives one answer and no indication of how it would have responded to the same question with the options reordered. The position-dependent swing in accuracy between first and last placement is invisible in the output. The 54 % flip rate in the low-confidence tercile is invisible too, unless someone deliberately runs the permutations. The model does not announce that its judgment is contingent; it presents a choice as though it were settled. That is the shape of the deception the study documents — not a lie, but a performance of certainty that the underlying computation does not support.
The Moment of Clarity
Understanding arrives when the flip rate stops looking like a quirk and starts looking like a measurement of how much of the model’s answer was ever about the question. At 25.8 % on the released checkpoint, a quarter of the answers were partly about the list. At 7.2 % after shuffled training, a fourteenth still were. At 0.5 % with the parallel layout, the number becomes small enough to treat as residue rather than signal. The distance between those three figures is the distance between what the model claimed to do and what it did — and it was measurable all along, by anyone willing to move the options around. The study is available as a preprint on arXiv (arXiv:2611.09803).