AI Governance for Healthcare AI Companies

YOUR AISYSTEMHealth system security reviewPayer vendor assessmentHTI-1 source attributesCalifornia SB 503 packageEU AI Act technical fileONE DETERMINATIONRECORDEvidenced once, reused everywhereAssessed Govern · Continuous compliance for AI

Short answer: Organizations that build artificial intelligence (AI) products for healthcare are assessed separately by every hospital, health system, and payer that buys from them, and again by each regulator whose rules apply. Assessed Govern discovers each AI system, maps it to the controls those obligations require, proves whether each control exists with evidence, and keeps that record current, so one documented system answers many reviews.

Key takeaways

  • The obligations that reach healthcare AI developers come mostly through the buyer, not directly from a regulator.
  • HTI-1 source attributes, California SB 503, Section 1557, and the EU AI Act ask for overlapping evidence in different formats.
  • No certification currently certifies an AI product as responsibly governed, so buyers verify vendors one at a time.
  • Governance evidence that is assembled per deal lengthens the sales cycle; evidence that is maintained continuously shortens it.

Healthcare AI developers answer to their buyers before they answer to regulators

Most AI regulation in healthcare is written for the organization that uses the system rather than the organization that built it. Section 1557 of the Affordable Care Act obliges covered entities to monitor patient care decision support tools for discrimination. The Joint Commission’s guidance on responsible AI use sets expectations for health systems. State utilization management laws bind plans and insurers. In each case the duty sits with the deployer.

Deployers discharge those duties by asking vendors for evidence. A health system that must monitor a tool for discriminatory effect cannot do so without subgroup performance data from the developer. A plan that must show its utilization management is auditable needs audit records from the vendor that supplied the model. The regulatory burden transfers through procurement, and it arrives as a questionnaire.

The result is an asymmetry that governance teams at healthcare AI organizations recognize immediately. One AI system is documented once and then evidenced repeatedly, against a different template for every buyer, at every renewal, and after every material model change.

What health systems ask AI vendors to produce

Healthcare buyer review has converged on a recognizable set of artifacts. The Health Sector Coordinating Council Cybersecurity Working Group published its Third-Party AI Risk and Supply Chain Transparency Guide in April 2026; Appendix G of that guide sets out 69 AI vendor assessment questions for procurement and governance, risk, and compliance teams. The categories it covers are consistent with what health system reviews now request:

  • A software bill of materials and an AI bill of materials identifying which features are AI capable and which are enabled.
  • Quality assurance, verification, and validation documentation, together with clinical validation studies where the system informs clinical decisions.
  • Training data lineage, inclusion and exclusion criteria, demographic representativeness, and whether protected health information was involved.
  • Performance and fairness results reported by subgroup, including race, sex, age, and language, rather than in aggregate.
  • Audit records at the inference level, capturing inputs, model version, outputs, confidence, reviewer identity, and whether the output was accepted, edited, or overridden.
  • A business associate agreement, a list of subprocessors with sub-business-associate agreements for each, and notice when that list changes.
  • Change management commitments covering advance notice, sandbox testing for major updates, and rollback.
  • A SOC 2 Type II report, and increasingly ISO/IEC 42001 certification or a HITRUST AI assessment.

Buyers also apply risk tiers. A system that produces a non-directive summary for a clinician who reviews it receives lighter review than a system whose output influences timing of care, diagnosis, or treatment. Vendors who can state their tier, and evidence the controls that tier requires, move through review faster than vendors who cannot.

The obligations that land on the developer

A smaller set of requirements applies to the organization that develops the system. These are the ones a vendor cannot defer to its customer.

Obligation What it requires of a developer Status
ONC HTI-1 Decision Support Interventions, 45 CFR 170.315(b)(11) Disclosure of 31 source attributes for each Predictive Decision Support Intervention, covering intended use, training data, fairness, external validation, quantitative performance, and ongoing maintenance, plus intervention risk management across validity, reliability, robustness, fairness, intelligibility, safety, security, and privacy. It binds developers of certified health IT directly, and reaches other vendors through their certified partners. In force. The criterion replaced 45 CFR 170.315(a)(9), which expired 1 January 2025. The HTI-5 proposed rule published in December 2025 would remove these provisions; it had not been finalized as of late September 2026.
California SB 503 Developers of clinical decision support systems must make reasonable efforts to identify known and foreseeable bias risks, take reasonable steps to mitigate them, and give deployers documentation on training data representativeness, performance and limitations, bias mitigation, data governance, intended outputs, and recommended monitoring, at initial sale or on request, and again on material updates. Effective 1 January 2027.
California AB 489 AI systems must not imply licensed clinical oversight through professional terms, post-nominals, or interface elements. This is a duty on the system’s developer. Effective 1 October 2025.
California AB 2013 Developers of generative AI made available to Californians must publish the types, sources, and purpose of training data, including whether it contains personal information. Posting due 1 January 2026.
FDA Predetermined Change Control Plans Organizations whose software is a regulated device may authorize specified future model modifications at the time of marketing submission through a description of modifications, a modification protocol, and an impact assessment. Without an authorized plan, a retrained model may require a new submission. Final guidance for AI-enabled device software functions. The separate January 2025 draft guidance on lifecycle management remains draft.
EU AI Act, Annex I AI that is a medical device, or a safety component of one, under the Medical Device Regulation or In Vitro Diagnostic Regulation is high-risk. Providers must operate a risk management system, data governance, technical documentation, logging, transparency to deployers, human oversight, accuracy and cybersecurity measures, a quality management system, post-market monitoring, and serious incident reporting. The medical device regulations did not receive a carve-out. 2 August 2028 for AI embedded in regulated devices. 2 December 2027 for standalone high-risk systems under Annex III.
HIPAA, as a business associate An organization processing protected health information on a covered entity’s behalf is directly liable under the Security Rule, must hold a business associate agreement, and must flow equivalent terms to every subcontractor in the chain, including model hosts and inference providers. The security risk analysis at 45 CFR 164.308(a)(1) must cover model hosting, inference, and prompt and output logging. In force. The Security Rule modernization proposed in January 2025 has not been finalized.

Organizations selling into payers carry a further set. California SB 1120 has applied since 1 January 2025 and reaches the vendors that plans contract with for utilization review, requiring that any tool base its determinations on the individual’s clinical circumstances rather than a group dataset alone, and that it be auditable, periodically reviewed, and non-discriminatory. Comparable laws take effect in Washington, Iowa, and Alabama during 2026, in Colorado, Georgia, and Utah on 1 January 2027, and in Illinois on 1 January 2028. Several of them impose periodic accuracy audits and annual reporting on AI-assisted denials, which are software obligations rather than policy obligations.

Which healthcare AI credentials a vendor can actually hold

Vendors frequently ask which certification will satisfy a hospital’s review. The answer depends on what each programme certifies, and two of the most frequently cited do not certify products at all.

Programme Certifies a vendor or product? Scope
Joint Commission Responsible Use of AI in Healthcare certification No. It certifies healthcare organizations, and states that it does not validate or certify individual products or tools. Governance, data management, risk and bias reduction, performance monitoring, and transparency and training at the organization that uses AI.
HITRUST AI Risk Management Assessment No. The output is an AI Risk Management Insights Report rather than a certification. 51 requirements covering AI risk management practices and governance, mapped to ISO/IEC 23894 and the NIST AI Risk Management Framework.
HITRUST AI Security Assessment and Certification Yes. It produces a shareable certification. 44 requirements covering the security and privacy of the AI platform. It can be issued standalone or alongside an e1, i1, or r2 assessment.
URAC Health Care AI Accreditation Yes, through a dedicated developer pathway. Data training and governance, pre-deployment testing, validation and evaluation, model drift handling, and disclosure to customers covering intended use and performance limitations.
ISO/IEC 42001 Yes, as a management system certification for the organization. An AI management system. It appears with growing frequency in healthcare requests for proposal.
CHAI applied model card Not a certification. It is a disclosure format. An open schema and public registry for model capabilities, limitations, technical details, validation results, and compliance-relevant information. Buyers increasingly request it by name.

Because no programme certifies that a given AI product is responsibly governed, health systems verify vendors individually. That is why the same evidence is requested repeatedly, and why maintaining it continuously is more efficient than assembling it per deal.

How Assessed Govern produces the evidence

Assessed Govern is built on the ARISE Framework™, a governance model from Assessed Intelligence that unifies AI, cybersecurity, privacy, and ethics governance across 128 controls and 953 prioritized requirements, mapped to more than 42 frameworks. For a healthcare AI developer, that mapping is the point: the same control, evidenced once, answers a buyer questionnaire, a certification requirement, and a regulatory obligation at the same time.

  • Discover each AI system. Assessment reads code, cloud configuration, and identity configuration rather than relying on a questionnaire, which produces an inventory that reflects what is deployed rather than what was remembered.
  • Determine what applies. Each system is classified, and the obligations that follow from that classification are mapped through the ARISE crosswalk, including the EU AI Act, NIST AI RMF, ISO/IEC 42001, SOC 2, HIPAA, and the healthcare requirements set out above.
  • Prove the controls exist. Every requirement carries an evidence state rather than an assertion, so a buyer question is answered from a record instead of from recollection.
  • Gate deployment. A governance gate in the continuous integration and delivery pipeline blocks a non-compliant model from reaching production, which is also the control a buyer is describing when it asks about change management and rollback.
  • Keep the record current. The output is a tamper-evident AI determination record. When a model is retrained, the record reflects it, which is what both a Predetermined Change Control Plan and a California SB 503 material update disclosure require.

Common questions

Do the HTI-1 source attributes still apply if HTI-5 removes them?

They applied as of late September 2026, when no final HTI-5 rule had been published. The proposed rule would delete the predictive decision support source attributes and the intervention risk management provisions from 45 CFR 170.315(b)(11). Removing the certification criterion would not remove the underlying duties, which persist under Section 1557 of the Affordable Care Act, HIPAA, and state law. Federal certification tested a system once for many buyers; without it, each buyer verifies separately, which increases the number of reviews a developer answers rather than reducing it.

We are not a certified health IT developer. Why is our EHR partner asking us for predictive DSI source attributes?

Because the certification obligation sits with the certified product, and your model supplies the output that product surfaces. The certified developer must populate the source attribute fields and maintain their accuracy, and it can only do so with information from you. The requirement reaches upstream vendors contractually rather than directly.

Is a SOC 2 Type II report enough to sell AI into a health system?

It is necessary and not sufficient. SOC 2 addresses the security of the service; it does not address training data provenance, subgroup performance, clinical validation, model change control, or bias mitigation, which are the questions healthcare AI review now asks. Organizations that present SOC 2 alone generally receive a follow-up questionnaire covering the remainder.

Can an AI vendor be Joint Commission AI certified?

No. The Joint Commission’s Responsible Use of AI in Healthcare certification applies to healthcare organizations and states that it does not validate or certify individual products or tools. A vendor cannot hold it. A vendor’s customers can, and their certification depends in part on evidence the vendor supplies, so the practical effect is a further evidence request rather than a credential a vendor can obtain.

Which HITRUST AI assessment do healthcare buyers accept?

The HITRUST AI Security Assessment produces a shareable certification covering 44 requirements on the security and privacy of the AI platform. The HITRUST AI Risk Management Assessment covers 51 requirements on AI risk management practices and produces an insights report rather than a certification. A buyer asking for proof will accept the former as a credential and read the latter as supporting documentation.

Do we need a Predetermined Change Control Plan if we retrain our model?

Only if the software is a regulated device. Where it is, an authorized plan allows specified modifications to ship without a new marketing submission, provided the change falls within the description of modifications and the modification protocol that FDA reviewed. Where the software is not a device, no plan is required, but buyers still ask how model changes are validated and communicated, and the same documentation answers both.

What does California SB 503 require us to hand our hospital customers before 1 January 2027?

Developers of clinical decision support systems must provide deployers with summaries of training data types including demographic representativeness, an account of how performance, limitations, and bias mitigation were evaluated, the data governance measures and dataset suitability checks applied, the intended benefits and expected outputs, known and foreseeable bias risks with their mitigations, and recommendations for use, monitoring, and risk mitigation. The package is due at initial sale or on request, whichever comes first, and again on material updates. Its contents overlap substantially with the HTI-1 source attributes and the CHAI applied model card.

Are we a business associate if our AI only processes de-identified data?

Data that meets the HIPAA de-identification standard is not protected health information, and processing it alone does not create business associate status. The question is usually whether identifiers are genuinely absent at every stage, including in prompts, logs, and support access. Where protected health information is processed at any point, a business associate agreement is required, and equivalent terms must flow to each subcontractor that processes it, including model hosting and inference providers.

When does the EU AI Act apply to our AI medical device?

2 August 2028 for AI that is a medical device or a safety component of one under the Medical Device Regulation or In Vitro Diagnostic Regulation, which the AI Act treats as high-risk through Annex I. Standalone high-risk systems under Annex III apply from 2 December 2027. Transparency obligations under Article 50 applied from 2 August 2026. These dates follow the 2026 amendments, which moved the high-risk deadlines later than originally legislated.

Who is liable when an AI system contributes to a poor outcome?

Liability allocation depends on the jurisdiction, the contract, and the facts, and this is not legal advice. The practical pattern is that deployers carry duties of oversight and monitoring while developers carry duties of disclosure and validation, and contracts allocate the remainder. Documentation determines which side an allocation falls on, because a developer that disclosed a known limitation stands differently from one that did not.

Join the beta

Tell us about your organization. We use these details only to contact you about the beta, as described in our Privacy Policy.

Beta offer: 30% off, held for two years

Organizations accepted into the beta lock in 30% off list price. That rate holds for two years from general availability.

Interested in having a conversation? Send a message to info@assessedsolutions.ai.

By signing up, you agree that Assessed Solutions may contact you about the beta. You can ask us to delete your details at any time. See our Privacy Policy.

You're on the list.

Thank you for your interest. We will contact you at the email you provided with next steps for the beta.