Medcurity Logo
Login

Can You Put PHI Into an LLM? HIPAA Rules for Using AI With Patient Data

Large language models have moved from novelty to daily workflow inside healthcare organizations — drafting patient messages, summarizing charts, answering coding questions. The single most common question we hear from compliance officers in 2026 is deceptively simple: can we actually put protected health information (PHI) into one of these tools? The short answer is that it depends entirely on which tool, which tier, and whether a Business Associate Agreement (BAA) is in place. This guide walks through what HIPAA requires before PHI touches an LLM, which vendors will sign a BAA, and how to build a defensible policy instead of a quiet workaround.

The short answer: only under a signed BAA, and only on a covered tier

Under HIPAA, any vendor that creates, receives, maintains, or transmits PHI on your behalf is a Business Associate, and you must have a BAA with that vendor before disclosing PHI to it. An LLM provider is no different from a cloud host or an EHR vendor in this respect. If you paste PHI into a chat interface that is not covered by a BAA, you have made an impermissible disclosure — regardless of how useful the output was. So the gate is not “is this AI smart enough,” it is “will this vendor sign a BAA, and does the tier I’m using fall under it.”

Which LLM vendors will sign a BAA (and which tiers are covered)

As of 2026, the major model providers will enter a BAA — but almost always only for their enterprise or API products, never their free consumer chatbots:

Two cautions. First, a vendor offering a BAA does not automatically mean every feature is in scope — training opt-outs, logging, and specific model endpoints can each carry conditions. Read the BAA and the product’s HIPAA documentation together. Second, a signed BAA is necessary but not sufficient: you still owe a vendor risk assessment on the provider, because a signed BAA is not the same as verified security.

Why consumer chatbots are off-limits for PHI

Free consumer LLM tools typically reserve the right to use submitted content to improve their models, retain conversation history under consumer terms, and provide no BAA. Each of those facts, on its own, makes them unsuitable for PHI. The practical risk is not hypothetical: staff who quietly paste a patient summary into a personal ChatGPT account to “save time” are creating exactly the kind of ungoverned exposure that surfaces during an OCR investigation. This is the core of the shadow-AI problem in healthcare — and it is a governance failure, not a technology failure.

The minimum-necessary and de-identification levers

HIPAA’s minimum-necessary standard applies to AI prompts just as it does to any other disclosure: send only the data the task actually requires. Two further options reduce risk substantially. De-identification under the Safe Harbor method (removing the 18 specified identifiers) or Expert Determination takes data out of PHI status entirely — de-identified data is not subject to the BAA requirement, though re-identification risk must be genuinely controlled. Limited data sets under a data use agreement are a middle path for research and analytics. Where a workflow can run on de-identified inputs, that is almost always the cleaner design than routing full PHI through a model.

Where this fits in your HIPAA risk analysis

Introducing an LLM into a clinical or administrative workflow is a change to your environment, and your HIPAA risk analysis should reflect it. That means inventorying which AI tools are in use, where PHI flows into them, what safeguards and BAAs cover them, and what residual risk remains. The proposed 2026 updates to the HIPAA Security Rule — still a Notice of Proposed Rulemaking as of mid-2026, not a final rule — would push organizations toward exactly this kind of asset inventory and technology-mapping discipline. Building it now, ahead of finalization, is the low-regret move. Medcurity’s guided Security Risk Analysis is $499/yr and includes the vendor and technology inventory that an AI-enabled practice needs.

A practical checklist before PHI meets an LLM

  1. Confirm a signed BAA is in place with the specific vendor and product tier.
  2. Verify the tier you use is actually in scope (API/enterprise, not consumer).
  3. Turn off model-training on your data and confirm retention/logging terms.
  4. Apply minimum-necessary — or de-identify — before the prompt is sent.
  5. Run a vendor risk assessment; a BAA alone is not verification.
  6. Write the tool into policy and your risk analysis, and train staff so no one is forced into a shadow-AI workaround.

Used deliberately, LLMs are compatible with HIPAA. The failures come from skipping the BAA, using the wrong tier, or letting AI adoption outrun governance. Talk to Medcurity about building an AI-ready HIPAA program that documents every tool touching PHI.

Frequently asked questions

Is it a HIPAA violation to put PHI into ChatGPT?

Entering PHI into the free or Plus consumer versions of ChatGPT is an impermissible disclosure because those tiers are not covered by a BAA. PHI may be used with OpenAI’s API or ChatGPT Enterprise only when a BAA is signed and the configuration meets HIPAA safeguards.

Which AI companies will sign a HIPAA Business Associate Agreement?

As of 2026, OpenAI (API and Enterprise), Anthropic (API/commercial), Google (Vertex AI / Gemini API on Google Cloud), and Microsoft (Azure OpenAI Service) will sign BAAs for their enterprise or API products. None of their free consumer chatbots are covered.

Can we avoid the BAA requirement by de-identifying data first?

Yes. Data de-identified under HIPAA’s Safe Harbor or Expert Determination methods is no longer PHI, so the BAA requirement does not apply — provided re-identification risk is genuinely controlled. De-identifying inputs is often the cleaner design for AI workflows.

Does using an LLM change our HIPAA risk analysis obligations?

Yes. Adding an AI tool that touches PHI is an environment change that your risk analysis should capture, including the data flows, safeguards, and BAA coverage for that tool. The proposed 2026 Security Rule updates reinforce this inventory-and-mapping expectation.