HIPAA Compliant LLM: The 4 Ways to Run One (and How We Help You Pick)

There is no HIPAA compliant LLM on a shelf. There are HIPAA compliant LLM deployments: a model served through an endpoint covered by a BAA, or on hardware you operate, wrapped in the controls the Security Rule expects. There are exactly four architectures that work. We build all four.

If you landed here from Definite's excellent guide to the same four architectures, you are in the right place. That post is the clearest technical survey of the space I have found. This one is the builder's companion to it, written from the other side of the table: the fractional CTO who has to make one of those four actually run, pass review, and stay compliant when the auditors ask questions.

At Empowered.guru we do this work for healthcare, accounting, and legal teams. Much of our consulting lands on architectures 3 and 4, the two where the model runs on private GPUs you control, either rented and dedicated in your VPC or on prem in your own data center, because that is where PHI stays tightest and where professional privilege has the least ambiguity. But we advise and ship across all four, because the right answer depends on your risk posture, not our favorite stack.

The ground truth before you pick

Three constraints apply no matter which of the four you choose. If these are not handled, the architecture does not save you.

There is no certification to buy. Since HHS issued its FAQ stating no HIPAA certification exists for any software, any vendor selling a certified LLM is selling vocabulary. What exists are compliant deployments inside organizations doing their own HIPAA work.

The BAA chain has to be unbroken. Every party that creates, receives, maintains, or transmits PHI on your behalf needs a signed BAA: the model provider if one exists, the cloud provider if PHI touches its infrastructure, and any tool in between. One uncovered hop makes traffic through it an impermissible disclosure.

The Security Rule follows the prompt. Encryption in transit and at rest, role based access control, and audit controls that record who prompted what, when, and what returned. Your inference logs are now compliance artifacts. Retention settings for those logs are policy decisions, not engineering defaults.

Compliance also stays shared. A BAA covers the vendor's side. Risk analysis, workforce training, minimum necessary policies, and configuration discipline stay on yours. AWS, Microsoft, and Google state this explicitly in their HIPAA materials. Take them at their word.

If you came here wondering whether a specific chatbot is allowed for PHI, the short answer is not in consumer mode. OpenAI, Anthropic, and Google offer covered configurations only through specific API surfaces described below. ChatGPT, Claude chat, Gemini chat, and Copilot chat in their normal consumer forms are not that surface.

Architecture 1: a cloud model endpoint under your cloud BAA

What it is. Frontier models as a managed service inside your cloud account. The three to know are Amazon Bedrock on AWS, Azure OpenAI under Microsoft's BAA via the Online Services DPA, and Gemini on Google Cloud under the Google Cloud BAA with a covered services list. All three are explicitly listed as HIPAA eligible by their cloud, which is the gate that matters.

Who signs the BAA. Your cloud provider, under the agreement you almost certainly already have. No new business associate enters your risk register.

What it demands operationally. Region pinning for residency. Wiring the endpoint's logs, like CloudTrail for Bedrock, into your audit story. Confirming no training defaults in the cloud agreement. IAM scoping so only the PHI approved workload can invoke the endpoint. The honest caveat is that prompts still process inside the provider's managed service. This is inside your cloud boundary, not on your silicon.

Choose it when you want frontier quality with the least new paperwork. For most healthcare teams starting out, this is the default we recommend while you prove the use case. It also keeps the door open to shift to a private stack later without changing the product.

How we help with it. We stand up the endpoint, lock the allowed surface in code review, and connect it to your existing VPC, identity, and logging so the model call inherits the controls you already run. If you already live on Bedrock or Azure OpenAI, we often keep you there for the first iteration.

Architecture 2: dedicated capacity or a vendor API with its own BAA

What it is. Going to the model vendor directly under its healthcare terms. The two to know are OpenAI's zero retention API under the Healthcare Addendum and Anthropic's HIPAA ready first party API. For throughput or isolation, both clouds also sell dedicated capacity, provisioned throughput on Bedrock and provisioned deployments on Azure OpenAI, that keeps the same BAA story as architecture 1 with single tenant performance.

Who signs the BAA. The model vendor, which means a new business associate and a vendor risk review.

What it demands operationally. Narrow configuration discipline. The covered surface is smaller than the full product. OpenAI's BAA covers only zero retention eligible endpoints on an approved org, with third party GPTs and plugins excluded. Anthropic's covers a defined subset of the Messages API. Feature drift is the failure mode: an engineer turns on an excluded capability and your covered deployment quietly stops being covered. We pin the allowed surface in code, not in a wiki, and gate deploys on it.

Choose it when you need a vendor specific capability the cloud endpoints do not expose, or you need throughput guarantees that shared endpoints cannot promise.

How we help with it. We scope the vendor review, draft the allowed surface as a checked policy, and keep first party and cloud endpoints swappable behind the same interface so you are not locked to one vendor's roadmap.

Architecture 3: open weights, self hosted in your cloud

What it is. Serving open weights models yourself on GPUs you control in your VPC, behind a serving stack like vLLM or similar. Current workhorses are in the Qwen, DeepSeek, Mistral, and Llama families. No model vendor ever sees a prompt.

Who signs the BAA. Only your cloud provider, because PHI still lives on its infrastructure. There is no model BAA because there is no model vendor.

What it demands operationally. Everything the previous two hid. You own TLS termination, authentication in front of the endpoint, audit logging of every request, patching the serving framework, autoscaling, and capacity planning. No training promises become trivial, nobody is training on your prompts, but retention becomes your code. If your gateway logs prompts in plaintext to a broad log store, you built the leak yourself. The good news is that with a structured system around the model, current open weights handle analyst style work well. See OpenAI's enterprise notes on model routing and the broader shift toward harness, loop, and graph patterns that matter more than raw frontier gap for structured tasks.

Choose it when policy forbids any model vendor from seeing PHI, you want token spend to flatten into a GPU line item, or you need portability across clouds.

How we help with it. This is the center of our PHI practice. We run dedicated, single tenant GPU stacks in your VPC, either on your reserved capacity or on rented dedicated GPUs we provision for you, with vLLM, autoscaling, and a private gateway you control. No shared inference pool. We wire audit logging, access control, and retention so the Security Rule story is complete. For CPA and law firms we apply the same pattern to engagement files and matter data where the concern is PII and privilege rather than PHI, but the controls rhyme. This is also where we do the most tuning, benchmarking open weights against your tasks before you pay for a bigger model you may not need.

Architecture 4: on prem GPUs

What it is. Open weights on hardware you own, in your data center. The air gap tier.

Who signs the BAA. No one in the inference path. This is the only architecture with no third party between the prompt and the weights, which makes it the only true no BAA required answer for inference. Any cloud variant of self hosted still requires the cloud provider's BAA, because the infrastructure holding PHI is theirs. Worth repeating in the other direction: the moment PHI rests on someone else's disks, you need their BAA even if the model is open and the stack is yours.

What it demands operationally. Everything in architecture 3 plus hardware: procurement, racking, power and cooling, failover, and physical safeguards, which the Security Rule also covers. Model updates arrive on your schedule, a feature for change control and a tax on capability. Egress can go to zero. For many clients we pair a small on prem cluster for the highest sensitivity workflows with a VPC hosted cluster for burst, with routing rules that keep the sensitive slice local.

Choose it when you are a hospital system, payer, research lab, or a firm with an existing data center practice and a policy that says PHI, or privileged matter data, does not leave the building. Defense and public sector teams end up here for the same structural reasons.

How we help with it. We design and run the on prem stack with the same private gateway and controls as the VPC version, including single tenant GPUs, encrypted volumes, and auditable logs that stay on your network. When clients already own GPUs, we bring them into service. When they do not, we spec and procure with you and operate the stack after install. We keep the VPC and on prem paths as one system, so a workflow can stay inside the building or burst to your VPC without changing the product.

The decision table

The checklist every build still owes

Architecture picks who you trust. These six items exist on every build we ship.

1. BAAs signed before the first PHI request with every party in the path. For accounting and legal, translate this to DPAs and engagement terms for PII and confidential matter data.
2. No training and retention in writing. Defaults are not commitments. Get the clause or, with open weights, remove the vendor from the chain entirely.
3. Audit logging of every prompt and response retained per policy and queryable when you need to answer OCR or a client.
4. Access controls that enforce minimum necessary. The model sees what the calling user is allowed to see, not what the service account can see.
5. Residency you can state in one sentence region by region and, for on prem, rack by rack.
6. An incident path that includes the model endpoint. Who gets paged, who is notified, and where the logs go when the endpoint misbehaves.

A model endpoint is not a product

The trap at the end of this project is the one Definite names clearly and we see just as often: you stand up a clean model endpoint, then someone asks if it can look at the actual data. Now a database, a semantic layer, a BI surface, and an agent runtime all enter the PHI path, and teams bolt those on as SaaS, reopening every boundary the endpoint just closed.

The model is one component. A real product is the system around it: connectors, a lakehouse or warehouse, access governed views, and an agent that calls the endpoint you chose without copying PHI into someone else's product. We build that system to live where your endpoint lives: in your VPC on dedicated GPUs or on prem, calling Bedrock, Azure OpenAI, Google, or your own open weights through one gateway, with the same audit and access story end to end.

Definite frames this as a private AI data analyst and a private deployment in your cloud. That framing is sound and our builds follow the same principle under our own stack, usually on your cloud with rented dedicated GPUs or on hardware you own. The buying lens for the wider space is well covered in Definite's HIPAA compliant AI tools guide, and their process view in the self hostable data stack.

If you want to evaluate the four with your data in the room, we will run a short architecture review and a time boxed pilot on the stack that matches your policy, not ours. For teams that start on architecture 1 or 2 and migrate toward 3 or 4, we keep the product and the data plane identical and move only the endpoint. That portability is the point. See how we scope this work in choosing your startup tech stack and what we ask for in technical due diligence.

FAQ

Is there a HIPAA compliant LLM? Not off the shelf. No model is certified and no HIPAA certification exists for any software. What exists are compliant deployments, a model served through a covered endpoint or on hardware you operate, inside an organization doing its own HIPAA work.

Is Amazon Bedrock HIPAA eligible? Yes, on AWS's HIPAA eligible services list. That lets you run PHI workloads through it once your AWS BAA is in place and the workload is configured per HIPAA. Azure OpenAI and Google's covered Gemini services have parallel coverage under their respective BAAs.

Can I use an open source LLM for HIPAA workloads? Yes. Serving an open weights model yourself means no model provider touches PHI. You still need your cloud provider's BAA if it runs in your cloud. Only the on prem variant has no third party in the path at all.

Can we avoid a BAA entirely? Only on hardware you own and operate in your own facility. The moment PHI is stored or processed on another party's infrastructure, that party is a business associate.

Do you handle accounting and legal, not just healthcare? Yes. We run the same private GPU patterns for CPA firms and law firms where the sensitivity is PII, financial data, or privileged matter data rather than PHI. The controls map closely, and the same four architectures apply with different agreements.

What does it cost? Architecture 1 and 2 bill per token. Architecture 3 bills per GPU hour and flattens with volume. Architecture 4 is capex plus operations. We size and quote after a brief scoping of your workloads and retention needs. See how much an MVP actually costs and fractional CTO cost in 2026 for how we price the surrounding product work.

This article is educational and not legal advice. HIPAA obligations turn on your risk analysis, policies, and configuration. If you need a compliance determination, involve your privacy officer and counsel alongside the technical build.

About the Author

I'm Brian Marvin, an AI-native Fractional CTO with 30 years in technical leadership. At empowered.guru, I help startups build MVPs, shape roadmaps, and make AI-powered technology decisions that scale.