Pharmacists Hold the Key to AI Healthcare Governance
The medication review begins the same way it has for decades. A pharmacist scans the patient’s chart: a new prescription for an anticoagulant, a history of kidney disease, a recent discharge from the hospital. The algorithm in the hospital’s system flags no interactions. But the pharmacist knows the patient is also taking an over-the-counter supplement that amplifies bleeding risk — information the model never saw. The pharmacist makes the call. The model, meanwhile, learned nothing from the decision it never knew it was making.
This is the hidden architecture of healthcare AI governance. It does not live in boardrooms or compliance checklists. It lives in the daily, high-consequence judgments of clinicians who have been practicing what looks exactly like algorithmic oversight for years — without anyone calling it that. The question is not whether healthcare can build AI governance. It is whether the industry will recognize the governance infrastructure already standing in front of it.
The Pharmacy as Governance Laboratory
Pharmacists occupy a strange position in the healthcare hierarchy. They are among the most rigorously trained professionals in the system — doctorates in pharmacology, years of clinical rotations, board certifications — yet they are often treated as order-fillers rather than decision-makers. This gap between capability and authority is precisely what makes them the ideal case study for AI governance.
Dan Eduardo Gonzalez, Pharm.D., founder of ClarityRx Advisory, argues that the profession has been running a real-time governance experiment for decades. Every medication review requires the pharmacist to weigh clinical evidence against safety protocols, regulatory requirements against operational realities, ethical obligations against patient preferences. These are not technical decisions. They are judgment calls made under conditions of uncertainty, with consequences that can mean life or death.
Consider the contrast with how technology companies approach governance. A Silicon Valley AI team might spend months debating fairness metrics, bias audits, and model cards. A pharmacist, by contrast, makes these same trade-offs in seconds, while also managing a queue of patients, answering phone calls from physicians, and ensuring that the pharmacy’s inventory meets regulatory standards. The pharmacist does not have the luxury of theoretical debate. Governance, for them, is not a framework. It is a workflow.
This operational grounding is precisely what AI governance frameworks lack. A 2024 global tech report cited by Gonzalez found that 86 percent of surveyed healthcare organizations are embedding AI into workflows, but the same report noted that regulatory complexity and oversight remain significant barriers. [1] The gap is not technological. It is translational. The people building the models do not understand how clinical decisions actually happen, and the people making those decisions are rarely invited to the table where the models are designed.
The Cost of Late-Seat Involvement
Gonzalez recounts a telling interview with an AI healthcare company. He asked a question that seems obvious in retrospect: “Where does the clinician sit relative to the engineers — before the build, during, or after?” The company interpreted this as a sign that Gonzalez was not interested in being an individual contributor. They missed the point entirely. The question was not about his role. It was about the structural position of clinical expertise in the development process.
This confusion reveals a fundamental misunderstanding about what governance requires. Most organizations treat clinicians as validation tools — brought in after the model is built to check for obvious errors, then sent back to their clinical duties. This is like asking a pilot to approve an aircraft design after the wings are already attached. The pilot can tell you the plane will crash, but they cannot tell you why, because they were not present for the aerodynamic decisions that made the crash inevitable.
The economic implications are stark. Gonzalez notes that for every hour of “unproductive labor” per employee per week, the average annual cost to a company is approximately $15,000. When clinicians are brought in only at the validation stage, they spend their time identifying problems that could have been avoided with earlier involvement. The rework cascades across teams: engineers rewrite code, compliance officers redo impact assessments, executives reallocate budgets. The cost of late-seat involvement is not just inefficiency. It is the systematic erosion of trust between the people who build AI and the people who must use it.
The International Governance Gap
Look at how other countries handle this relationship. In Germany, the Federal Institute for Drugs and Medical Devices requires that AI-based medical devices undergo a “conformity assessment” that includes clinical evaluation by practicing physicians. The clinicians are not consultants. They are auditors with binding authority. In the United Kingdom, the National Institute for Health and Care Excellence mandates that AI tools be evaluated against real-world clinical workflows, not just technical benchmarks. The evaluators are frontline staff, not data scientists.
The United States, by contrast, has no equivalent requirement. The FDA’s framework for AI/ML-based medical devices focuses on the algorithm’s performance characteristics — sensitivity, specificity, area under the curve — but says little about how the tool integrates into the actual decision-making environment of a busy clinic or pharmacy. The result is a regulatory landscape that rewards technical sophistication while ignoring operational reality.
This is where Gonzalez’s background as a bilingual practitioner becomes relevant. He bridges not just language but culture — the culture of clinical practice and the culture of technology development. In his experience, the most productive governance conversations happen when both sides are forced to translate their priorities into terms the other can understand. The engineer learns that “latency” means something different to a pharmacist managing a medication error. The pharmacist learns that “interpretability” is not the same as “explainability.” These translations are not trivial. They are the substance of governance.
The Unseen Governance Infrastructure
The argument that healthcare already possesses governance expertise is not a metaphor. It is a structural claim about where accountability actually lives in the system. When a pharmacist catches a drug interaction that the AI missed, they are performing a governance function: they are ensuring that the technology’s output does not override clinical judgment. When a nurse notices that the AI’s triage recommendations consistently underestimate the severity of certain symptoms in elderly patients, they are performing a governance function: they are identifying a systematic bias that the model’s training data failed to capture. When a physician refuses to override a patient’s documented preference because the AI’s recommendation does not account for it, they are performing a governance function: they are asserting that the model’s optimization objective does not supersede the patient’s autonomy.
These moments are happening thousands of times a day, in every healthcare setting, without being recognized as governance. They are invisible because they are routine. But routine is precisely where governance lives. The most dangerous failures are not the dramatic ones that make headlines. They are the quiet ones that become normal — the decision to trust the model when it should have been questioned, the workflow that slowly adapts to the algorithm’s limitations until those limitations become invisible.
Gonzalez’s insight is that healthcare does not need to invent new governance institutions. It needs to recognize the ones it already has, and then give them the authority and resources to function effectively. This means changing the question from “How do we govern AI?” to “Who is already governing AI, and are we supporting them?”
The Open Question
The conversation about AI governance in healthcare has been dominated by technologists and compliance officers. They ask about model cards, bias audits, regulatory filings. These are important questions. But they miss the deeper one: what happens when the governance infrastructure that already exists is ignored?
Consider the pharmacist who catches the AI’s error. They have no formal role in the AI governance framework. They are not consulted about model design. They are not included in deployment decisions. They are simply doing their job, as they have always done, applying clinical judgment to a situation where the technology failed. The system benefits from their expertise without acknowledging it, without learning from it, without incorporating it into the feedback loop that would make the next model better.
This is not sustainable. As AI becomes more embedded in clinical workflows, the gap between the governance that is visible and the governance that is real will widen. The visible governance will become more sophisticated — more metrics, more audits, more frameworks. The real governance will become more invisible — more silent corrections, more unrecorded overrides, more unshared insights. The two will drift apart until the invisible governance can no longer keep up with the visible technology.
The open question is whether healthcare organizations will recognize this drift before it becomes a crisis. Gonzalez’s work suggests that the answer depends on whether the industry can learn to see the governance that is already there. Not by inventing new roles, but by empowering the ones that exist. Not by building new frameworks, but by connecting the frameworks to the workflows that already govern clinical decisions. The pharmacist who caught the anticoagulant error did not need a governance framework to know that the model was wrong. They needed the model’s builders to understand why. That understanding cannot be added after the fact. It must be built into the process from the beginning, because the beginning is where the decisions that matter are made.
The pharmacist who caught the anticoagulant error did not need a governance framework to know that the model was wrong. They needed the model’s builders to understand why. That understanding cannot be added after the fact. It must be built into the process from the beginning, because the beginning is where the decisions that matter are made.