When SQL Queries Run But Return Wrong Answers
The Execution Illusion
A database query that returns a clean table of results looks like success. No error message. No exception thrown. The rows appear, the columns align, the numbers populate cells that seem to answer the question that was asked. This is the moment where more output stops meaning more truth.
Large language models now write SQL from natural language questions with increasing fluency, and the queries they produce often execute without complaint. [1] According to research on semantic errors in generated SQL, a query can be syntactically valid and run without an exception while still returning a wrong result, so execution alone does not establish correctness. The database does not know what you meant to ask. It only knows what you told it to do.
Why Diagnosis Without Direction Fails
An error taxonomy offers something valuable: it says why a query is wrong. Attribute mismatch. Missing join. Incorrect aggregation. These labels categorize failure. But a category is not a compass. Knowing that a query has an attribute mismatch does not tell you which column carries the error, which table should have been referenced, or what operation would fix it. The TEG paper makes this limitation explicit: an error taxonomy says why the query is wrong, but not where to look or how to change it. The diagnosis floats above the code, pointing at a problem without pointing at a location.
This is the mechanism that explains why so many correction methods miss their target. They receive a label and must infer everything else. The model gets told “this is wrong because of a mismatch” and is then expected to scan the entire query, find the faulty construct, understand what it should have been, and produce the fix. The diagnosis provides no bridge between the abstract category and the concrete edit.
The Architecture of Grounding
A method called TEG — Taxonomy-guided Error Grounding — takes a different approach. Rather than treating the error type as context that accompanies the query, it converts the diagnosis into a structured correction input. The taxonomy becomes a set of instructions.
The mechanism works through type-specific rules. Each error type maps to two things: construct classes to reconsider, and an edit operation to request. When the error involves an incorrect reference, the rule selects expressions that carry references and requests a replacement. When something is missing entirely, the rule selects no existing element and requests an insertion.
This mapping is what the researchers call error grounding: translating a taxonomy-level diagnosis into the classes of query constructs to reconsider and the edit operation to request. The diagnosis stops being a label and becomes a directive.
The implementation parses the query into a syntax tree, making these construct classes addressable. TEG then masks the selected nodes when applicable and states the operation in an edit instruction.
The Mask as Instruction
Masking does something subtle. It removes information from the input, but the removal itself carries meaning. When a column expression disappears from the query, the model is not left wondering what to do — the edit instruction tells it. The absence marks the location. The instruction specifies the action.
This inverts the usual relationship between context and correction. Instead of providing more information and hoping the model finds the problem, TEG provides less information in a specific place and tells the model what kind of fix belongs there. The constraint becomes the guide.
The approach acknowledges a tension: because the rules select construct classes rather than exact fault locations, masks can cover correct constructs. The model receives a query with a hole where something used to be, and that hole might have contained working code. The model must generate a complete query from the grounded input, potentially rewriting more than the actual error required.
This imprecision is a deliberate trade. Exact fault localization is hard. Construct-class selection is tractable. The method accepts that some correct code may be disturbed in exchange for reliably directing attention to the right region.
When One Error Becomes Many
Real queries often contain multiple errors. A single prompt carrying every error type would mix several masks and instructions, creating confusion about which edit applies where. TEG processes one annotation at a time instead.
The sequence matters: editing existing constructs before inserting new ones, then re-grounding the output of each step. Each correction becomes the input for the next diagnosis. The query evolves through a series of grounded edits rather than a single comprehensive rewrite.
This sequential approach reflects how the errors relate to each other. An incorrect reference might need to be fixed before a missing join can be properly inserted. The order of operations is not arbitrary — it follows the dependency structure of the query itself.
What the Numbers Show
On NL2SQL-BUGs, a benchmark of incorrect SQL queries derived from the BIRD dataset, TEG reaches 47.3 percent execution accuracy on single-error queries and 37.0 percent overall when paired with Qwen2.5-7B-Instruct. [1] The lead holds across model sizes and thinking modes evaluated in the main comparison. TEG outperforms every evaluated baseline on single-error queries, even when those baselines receive the same error-type annotations. The advantage is not from better diagnosis — it is from better use of the same diagnosis.
When the error types themselves are predicted rather than supplied, TEG stays above direct LLM correction and ErrorLLM on single-error queries. [1] The grounding mechanism does not require perfect inputs to provide value.
The masking-based editing configuration within TEG contributes measurably to the result: removing it while keeping the error types lowers single-error accuracy from 47.3 to 40.9 with Qwen2.5-7B-Instruct. [1].
Beyond SQL
The framing extends beyond database queries. Large language models now generate code from natural language specifications across many domains. A generated program can be syntactically valid and run without an exception while still returning a wrong result.
The Limits of Self-Correction
Research on self-correction notes that improvement through intrinsic feedback alone is not guaranteed. The model can talk itself into confidence about a wrong answer. The critique sounds reasonable. The explanation is coherent. The fix does not work.
Taxonomies as Infrastructure
TEG’s contribution is the specific mapping from error type to construct class and edit operation. The taxonomy does not just label the problem — it determines the shape of the correction input.
The Verification Gap
This gap between running and being right is where the risk lives. Not a lie told by the model, but a structural property of how we verify machine-generated code. We ask: did it work? The query ran, so we assume yes. The assumption is the problem.
The error type determines what to mask. The mask creates a specific kind of absence. The edit instruction specifies what operation to perform. The model generates from this structured input rather than from a query annotated with suggestions. That is the whole move: the diagnosis stops describing the problem and starts shaping the fix.