HIPAA and AI chatbots: what counts as PHI in a prompt?

A nurse pastes a discharge summary into ChatGPT to turn it into plain language for a patient’s family. A billing specialist pastes a denied claim to draft an appeal. A clinic manager pastes a week of appointment notes to summarize for a handoff. Each of these is a normal, well-intentioned use of an AI tool — and each one, if the text identifies a patient, is potentially a disclosure of protected health information to a vendor with whom the organization has no business associate agreement. That combination is how ordinary workdays turn into reportable HIPAA incidents.

This post walks through what actually makes prompt text PHI, why the BAA question is decisive, and what controls make AI tools usable in a covered entity. It is general information, not legal advice — your compliance counsel owns the final word.

PHI is broader than most people think

PHI is not "medical records." It is any individually identifiable health information held or transmitted by a covered entity or business associate — health status, care received, or payment for care, linked to an identifier. HIPAA’s de-identification safe harbor enumerates 18 identifier types, and the list is famously wide: names, all geographic subdivisions smaller than a state, all elements of dates directly related to an individual (birth dates, admission and discharge dates), phone numbers, email addresses, SSNs, medical record numbers, health plan and account numbers, device identifiers, URLs, IP addresses, biometric identifiers, full-face photos — and a catch-all for any other unique identifying characteristic or code.

Two consequences follow for prompts. First, removing the name is not enough: "72-year-old male admitted 3/14 with CHF, MRN 448-2291" is PHI three times over with no name in sight. Second, context identifies: in a small town or a rare-disease clinic, "my patient with the pancreatic transplant last month" can identify a person as surely as an MRN. If prompt text pairs health information with anything on that list — or anything that functions like it — it is PHI.

A prompt is a disclosure

HIPAA regulates uses and disclosures of PHI. Typing PHI into a third-party AI tool transmits it to that third party’s servers, where it is stored under their retention policy and, on some consumer tiers, eligible for model training by default. That is a disclosure in the ordinary regulatory sense, and it does not fit the treatment, payment, or operations permissions when the recipient is a consumer chatbot vendor with no agreement in place.

The employee’s intent — summarizing, drafting, translating — does not change the analysis. Neither does deleting the chat afterward: the disclosure happened at submission.

The BAA question decides almost everything

A covered entity can let vendors handle PHI when a business associate agreement makes the vendor contractually responsible for safeguarding it. So the operative question for any AI tool is simple: is there a BAA covering this specific product and tier?

For consumer AI chat products — free or individual-subscription ChatGPT, Claude, Gemini, Copilot, Perplexity — the answer is no. Some AI vendors will sign BAAs for particular enterprise or API offerings under particular configurations, and that landscape shifts quarter to quarter; the details are between your counsel and the vendor. The practical rule for the workforce is much simpler and much more stable: no BAA, no PHI — and assume the tab your staff has open is not the covered one.

De-identify before the prompt and the problem disappears

Properly de-identified data is not PHI, and HIPAA does not restrict it. That is the sanctioned path to getting AI’s value on clinical text: strip the 18 identifier categories first, then let the model summarize, draft, or translate what remains. "Draft a plain-language discharge explanation for an elderly patient with CHF on these medications" needs no identifiers at all to be useful.

The catch is that de-identification-by-hand fails at exactly the moments HIPAA incidents happen — busy people pasting long documents under time pressure. An admission date survives in paragraph four; an MRN hides in a header. The control has to be mechanical, and it has to run before submission.

Practical controls for covered entities

• Write the rule into policy: no PHI in any AI tool without a BAA covering that exact product and tier. Name the tools staff actually use.
• Enforce at the browser, before the prompt is sent. ShieldWays detects PHI — alongside PII, secrets, and financial data — on-device, and can warn, redact, or block per policy on ChatGPT, Claude, Gemini, Copilot, and Perplexity. Redaction is the workhorse here: identifiers come out, the clinical question still gets asked.
• Keep evidence without creating new PHI. Detection and enforcement events land in a content-free audit log — category, action, site, timestamp — which gives compliance a record of the control operating without copying patient data anywhere new. (Detection running on-device matters for the same reason: prompt text is never shipped elsewhere for scanning.)
• Train with real scenarios — the discharge summary, the claims appeal — not abstract slides, and pair training with the in-the-moment warnings that actually change paste behavior.
• Prepare the incident path: an accidental PHI prompt triggers the same breach risk assessment as any other unauthorized disclosure, so make reporting fast and blame-light.

AI tools and HIPAA are not incompatible — but the compliance boundary sits at the prompt box, upstream of everything the vendor does. Put the control there, make de-identification automatic instead of aspirational, and clinical teams can use these tools for what they are genuinely good at without each paste being a small act of regulatory faith.