INTERVIEW FORMAT A system that reports its own good behavior is making a claim about itself, and in medical technology that claim can go untested for as long as nobody checks. The AI model that scores its own confidence, the internal tool that passes document review, the quality dashboard that stays green — each presents a claim about what it is doing. Attrayee Chakraborty, a senior quality systems engineer at Analog Devices, frames the compliance problem around exactly this question: whether a company can explain what its system is supposed to do, what could go wrong, and how it knows its controls work. [1] The distance between those answers and the system’s actual behavior is where the real risk sits.
The Confidence That Is Not Evidence
The most consequential gap in medical AI is not a model that malfunctions loudly. It is one that performs plausibly while its underlying reasoning or data has quietly drifted. Chakraborty’s framing of the AI Act’s medical-product pathway turns on a similar distinction: high-risk classification depends on both the relationship to a regulated product and whether that product requires third-party conformity assessment, and simply using AI in a medtech company does not answer that question. The appearance of being regulated is not the same as being regulated.
This is where the assumption takes hold. A team assumes its quality management system covers AI because the system has always covered software, and the assumption holds until an auditor asks a question the existing processes were never designed to answer. The organization assumed its own compliance, and the assumption survived because nothing tested it. Chakraborty warns against both extremes — assuming the current QMS covers everything, or building a completely separate AI governance structure — and describes the middle path as keeping the processes that work while adding the controls that are genuinely missing. [1]
Three Lenses, One Blind Spot
The EU AI Act, the Medical Device Regulation, and ISO/IEC 42001 all push organizations toward the same foundational questions, but they approach them from different angles. That overlap is often presented as an efficiency opportunity. It is also a place where a gap can hide, because each framework can be satisfied on paper while the underlying uncertainty remains unexamined. A developer treats model retraining as a technical improvement; quality sees a change that needs validation; a regulator may need to assess whether it affects the conformity assessment. None of those perspectives is wrong, but they must converge before the change is made, not after.
The deception here is structural rather than malicious. When three compliance regimes are mapped onto one process, the resulting documentation can look complete while the hardest question — who decides whether a model is still performing acceptably — goes unassigned. Chakraborty lists that question among the AI-specific checks an existing system must be able to answer, alongside data ownership, supplier changes, and whether the people affected by the system have been considered at all. A certificate under ISO/IEC 42001 can support a strong governance program, but it does not replace the work of determining which legal requirements apply to the product.
The Misconception That Travels Fastest
Among the misunderstandings Chakraborty identifies, the most widespread is that every AI tool used by a medical-device company is automatically a high-risk medical-device AI system. [1] The corrective is to look at what the system actually does and its intended purpose. AI inside a diagnostic device and an internal tool that helps someone draft a document both need thoughtful oversight, but not necessarily the same regulatory pathway. The difference between those two cases is precisely where over-classification and under-classification both become possible — and where a company can mistake the label for the substance.
Over-classification has its own deceptive quality. It produces the appearance of rigor: more documentation, more review, more sign-off. What it does not necessarily produce is a better answer to whether the system works. The same is true of the opposite error, where an internal quality investigation tool is waved through because it is “just internal.” Chakraborty’s advice is to start with the use case rather than three separate regulatory checklists: establish where AI is being developed or used, its intended purpose, who relies on its outputs, and what could happen if it fails. Only then do the applicable obligations become visible.
Where the Time Actually Goes
Compliance is often blamed for slowing innovation, and testing, documentation, and review do take time. But Chakraborty draws a line between time spent building confidence in a product and time lost because nobody agreed on expectations early enough. A team that reaches the end of development before asking what evidence quality or regulatory needs may have to repeat testing, revisit data, or redesign a control. The schedule slip is real, but its cause is not the requirement — it is the late arrival of a question that was always going to be asked.
The same pattern appears again here. Teams that treat compliance as a launch-gate obstacle tend to schedule it as one, and it becomes one. Teams that agree upfront on intended use, acceptable risks, the evidence needed, and the changes that would trigger another review give themselves room to innovate without constantly wondering whether they are moving in the wrong direction. The difference is not the regulatory burden. It is whether the burden was anticipated or discovered.
Who Owns the Performance
Responsibility for whether an AI system actually performs as claimed is easy to distribute and hard to place. Chakraborty assigns distinct roles: research and development should be clear about the problem being solved and what a successful outcome looks like; technical teams should explain how the system works, what data it relies on, and where it can fail; quality should translate that into a process that can be validated, documented, monitored, and maintained; regulatory should be involved early enough to guide the pathway rather than brought in at launch. The people building the system, she argues, need to own its performance and understand its limitations.
That allocation matters because the distance between appearance and behavior often falls to no one in particular. The questions Chakraborty recommends bringing teams together around — what are we approving, what evidence supports it, what uncertainty remains, who is responsible if performance changes — are not compliance formalities. They are the terms on which a system’s claims can be tested rather than trusted.
The Detail That Shows the Distance
One of the AI-specific questions Chakraborty raises is also the least technical: have we considered the people affected by the system, not just whether the software meets its specification? Meeting the spec is the appearance. The effect on the people downstream is the behavior.
That is where the practice sits relative to the promise. Organizations can now describe AI governance in sophisticated terms — responsibilities, risk management, data governance, lifecycle controls, continual improvement — and still have no answer for who decides when a model has stopped being acceptable. Chakraborty’s view is that the frameworks ask the right questions. Whether anyone is positioned to hear the answer is a separate matter, and it is the one that determines whether a confident system is telling the truth.
