Software Architecture Decisions Now Extracted by AI from Code Changes
Software systems age like cathedrals built by rotating crews of masons who never leave a blueprint behind. Every renovation makes sense to the person holding the hammer at that moment, but the original reasoning — why this load-bearing wall sits here, why that corridor bends unexpectedly — evaporates with each departure. Architects call these ghost decisions Architectural Design Decisions, and for decades, recovering them meant interrogating whoever was still around or digging through code like an archaeologist sifting rubble. A new study suggests that job is quietly migrating from human memory to machine inference.
The Rationale Gap
The paper in question put four large language models to work on a task that has traditionally required deep institutional knowledge: extracting the “why” behind structural software choices from raw commit messages. Thirty developer-written decisions from open-source projects served as ground truth, and the models — Gemini 3 Pro, DeepSeek R1, Kimi K2, and Qwen3 — were asked to reconstruct the reasoning from code changes alone. [1] On surface-level similarity metrics, they performed respectably, with BERT-F1 scores above 0.81 across the board. [2] Few-shot prompting nudged the best performer from 0.828 to 0.847, a modest but consistent improvement that suggests the models learn the expected format quickly.
But the numbers tell only half the story, and the manual review of the outputs reveals where the real deficiency lies. The generated design decisions were consistently too long, drifted toward implementation mechanics, and — most damningly — missed the rationale that makes an architectural decision worth recording in the first place. A human architect reading these outputs would recognize the shape of the answer without recognizing the thinking behind it. The models captured what was decided but not why it was decided, which is rather like transcribing a courtroom verdict while omitting the evidence and deliberation.
The Invisible Hand of Judgment
This distinction between capturing decisions and understanding them cuts to the heart of what architectural knowledge management has always been about. The field emerged because software projects kept failing not from coding errors but from forgotten context — engineers making changes that contradicted earlier choices because nobody remembered those choices existed. The traditional solution involved dedicated architects conducting interviews, reviewing documentation, and essentially performing oral history on the codebase. That work was slow, subjective, and dependent on people who had been around long enough to remember the early battles.
What the study demonstrates is that LLMs can now perform the first pass of that historical reconstruction at scale, processing thousands of commits in the time a human architect would need for a single conversation. The efficiency gain is real, and it raises an uncomfortable question: if machines can extract the skeleton of architectural reasoning from code commits, what remains for the human expert? The answer, according to the study’s findings, is the part that matters most — the judgment about what constitutes a genuine architectural decision versus a mere implementation detail, and the ability to articulate the trade-offs that shaped the choice.
The Limits of Pattern Recognition
The failure mode observed in the study deserves closer attention because it reveals something fundamental about how these systems operate. When the models generated overly long, implementation-focused outputs, they were not being lazy or careless — they were doing exactly what statistical pattern matching does best. Commit messages contain abundant information about what code changed and how, so the models dutifully reproduced that layer of detail. The rationale, by contrast, lives in the spaces between commits, in the conversations that never got written down, in the constraints that were so obvious to the original developer that they never seemed worth mentioning.
This is the same gap that has plagued every attempt to automate knowledge work, from expert systems in the 1980s to today’s generative models: the most valuable knowledge is often tacit, embedded in context that never makes it into any explicit record. The study’s authors acknowledge this limitation directly, noting that the generated decisions “miss the rationale behind the decision” and calling for “architecture-aware LLM systems” that might eventually bridge that gap. [3] But there is a deeper problem lurking beneath the technical challenge, one that no amount of fine-tuning will solve.
The Responsibility Question
If an LLM produces an architectural decision that misses the rationale, and a developer relies on that incomplete reconstruction to make future changes, who bears responsibility for the resulting design flaw? The study does not address this question, but it hangs over the findings like a shadow. When a human architect extracts a design decision, they bring their own experience to bear on the task, filtering what they find through years of accumulated judgment about what matters in software design. When a machine does it, the filter is whatever patterns existed in its training data — which may or may not align with the actual reasoning that shaped the code.
This matters because architectural decisions are not neutral facts waiting to be discovered; they are interpretations shaped by the interpreter’s understanding of what good design looks like. Two architects examining the same commit history might legitimately extract different decisions, depending on whether they prioritize performance, maintainability, or simplicity. The study treats the human-written ADDs as ground truth, but that ground truth is itself subjective — one developer’s architectural decision is another’s implementation detail. If we hand this interpretive task to machines, we are not just automating a chore; we are delegating a judgment call to a system that cannot explain its own reasoning.
The New Division of Labor
What the study ultimately maps is not the replacement of architectural judgment but its relocation. The mechanical work of scanning commits, identifying candidate decisions, and producing structured summaries is now demonstrably within machine capability, and organizations would be foolish to ignore that efficiency gain. But the study’s own results suggest where the human role persists: in distinguishing the architectural wheat from the implementation chaff, in supplying the contextual knowledge that never appears in code, and in making the final call about what deserves to be recorded as a decision worth preserving.
The authors position their work as preliminary, a first exploration of terrain that will undoubtedly be mapped more thoroughly in the coming years. Their honest reporting of the rationale gap — rather than cherry-picking the impressive BERT-F1 scores — is itself a model of how this kind of research should proceed. As these systems improve, the boundary between what machines can extract and what only humans can reconstruct will shift, but the study suggests it will not disappear entirely. The architects who survive this transition will be those who understand that their value lies not in reading commits but in the judgment they bring to what they find there.
Sources
1. Gemini 3 Pro
2. DeepSeek R1
3. Kimi K2
4. Qwen3