AI Governance in Healthcare: A Practical Framework for HIPAA-Covered Organizations

Artificial intelligence is already inside your organization — in ambient scribes, claims tools, patient-facing chatbots, and the consumer apps your staff quietly paste notes into. The question is no longer whether to allow AI, but how to govern it so that patient data, regulatory obligations, and clinical trust are protected. AI governance in healthcare is the set of policies, roles, risk assessments, and controls that let an organization adopt AI deliberately instead of by accident. This guide lays out what a workable program looks like for a HIPAA-covered entity or business associate, and how it connects to the rules already bearing down on the sector.

Why healthcare AI governance is now a board-level issue

Two forces are converging. First, shadow AI: employees use general-purpose tools that were never approved, never assessed, and frequently not covered by a Business Associate Agreement. When protected health information (PHI) flows into a consumer chatbot with no BAA and no zero-data-retention guarantee, that is a HIPAA violation regardless of how helpful the tool was. Second, regulators are moving. The HHS AI Strategy released in December 2025, the ASTP/ONC HTI-1 transparency requirements for predictive decision support interventions in certified health IT, and the proposed HIPAA Security Rule all point the same direction: organizations are expected to know what technology touches PHI and to manage its risk on purpose.

The OCR Notice of Proposed Rulemaking to modernize the HIPAA Security Rule — published in the Federal Register on January 6, 2025, with the comment period closing March 7, 2025 — explicitly names artificial intelligence (alongside quantum computing and augmented reality) as an emerging technology covered entities must account for in their risk analysis. As of mid-2026 no final rule has issued, but the proposal would also remove the long-standing “required versus addressable” distinction and make controls such as asset inventories, encryption, and multi-factor authentication mandatory. Governing AI is, in practice, an extension of the security risk analysis you already owe.

The four pillars of a healthcare AI governance program

1. Govern — establish ownership and policy

Name an accountable owner (often a cross-functional AI governance committee spanning compliance, security, clinical, and legal) and write a plain-language acceptable-use policy. The policy should state which tools are approved, what data may and may not be entered, when a BAA is required, and how staff request a new tool. This mirrors the GOVERN function of the NIST AI Risk Management Framework, the voluntary federal standard most healthcare programs build on. Governance without a named owner becomes a document nobody enforces.

2. Map — inventory every AI system that touches PHI

You cannot govern what you cannot see. Build and maintain an inventory of AI systems — vendor tools, embedded model features inside existing software, and unsanctioned consumer apps surfaced through staff interviews and network logs. For each, record the data it processes, whether a BAA is in place, the vendor’s training-data and retention posture, and the clinical or operational decisions it influences. This asset-mapping step is also where the proposed Security Rule’s asset-inventory mandate and your vendor risk process meet.

3. Measure — assess risk before and during use

Run an AI-specific impact assessment for each meaningful use case: data sensitivity, potential for biased or inaccurate output, patient-safety exposure, and downstream reliance on the result. High-risk clinical decision support deserves deeper scrutiny and human-in-the-loop safeguards than a back-office summarization tool. Document the assessment so it stands up in an audit, and re-measure when the model, the vendor, or the use case changes.

4. Manage — controls, monitoring, and incident response

Translate assessments into controls: BAAs and zero-data-retention configurations where PHI is involved, access restrictions, logging, staff training, and a defined path to retire a tool that fails review. Monitor continuously — AI risk is not a one-time sign-off — and fold AI incidents into your existing breach-response workflow so a model failure or unauthorized disclosure is handled like any other security event.

Where AI governance meets HIPAA obligations

For a HIPAA-covered organization, AI governance is not a parallel program — it is a lens on obligations you already carry. The Security Rule requires a risk analysis; AI is now part of that analysis. The Privacy Rule limits PHI use and disclosure; entering PHI into an unapproved tool can breach it. Vendor management requires BAAs; an AI vendor that processes PHI on your behalf is a business associate. The cleanest way to operationalize all of this is to treat each AI tool as a vendor and each use case as a documented decision, then route both through the same risk and agreement workflow you use for the rest of your stack.

Medcurity helps healthcare organizations bring AI into that workflow: cataloging the tools in use, capturing BAAs and vendor risk, and tying it all back to a current Security Risk Analysis. For related guidance, see our overviews of HIPAA-compliant AI tools, the question of whether ChatGPT is HIPAA compliant, third-party risk management in healthcare, and the 2026 HIPAA Security Rule update.

A 30-day starting point

You do not need a perfect program to make real progress. In the first month: publish a one-page acceptable-use policy, stand up an AI inventory and survey staff about the tools they actually use, identify any case where PHI is reaching a tool without a BAA and shut it down or remediate it, and name the owner who will keep the program alive. From there, layer in formal impact assessments and continuous monitoring. Governance compounds — the organizations that start now will be the ones ready when the final Security Rule lands.

Frequently asked questions

Does HIPAA specifically require AI governance?

HIPAA does not name “AI governance” as a standalone requirement, but its existing obligations apply directly to AI. The Security Rule requires a risk analysis of all systems that create, receive, maintain, or transmit electronic PHI — which includes AI tools — and the proposed 2025 modernization explicitly names artificial intelligence as an emerging technology to assess. Any AI vendor that processes PHI on your behalf is a business associate and needs a Business Associate Agreement.

What framework should a healthcare organization use for AI governance?

Most healthcare programs build on the NIST AI Risk Management Framework, a voluntary federal standard organized around four functions — Govern, Map, Measure, and Manage. It pairs well with HIPAA because it emphasizes inventorying systems, assessing risk, and assigning accountability, which map cleanly onto the Security Rule’s risk-analysis and asset-inventory expectations.

What is “shadow AI” and why is it a HIPAA risk?

Shadow AI is the use of AI tools that were never approved or assessed by the organization — for example, a staff member pasting patient notes into a free consumer chatbot. It is a HIPAA risk because PHI can be disclosed to a vendor with no Business Associate Agreement and no guarantee that the data will not be retained or used for model training, which can constitute an impermissible disclosure.

How do we start an AI governance program quickly?

Begin with four steps in the first 30 days: publish a one-page acceptable-use policy, inventory the AI tools your staff actually use, eliminate or remediate any case where PHI reaches a tool without a BAA, and assign a clear owner. Formal impact assessments and continuous monitoring can follow once the basics are in place.

Ready to bring AI into your compliance program instead of around it? Talk with the Medcurity team about governing AI alongside your Security Risk Analysis and vendor risk management.