Who Has to Prove Your Clinical AI Isn’t Biased? California Splits the Duty - Quarles

Who Has to Prove Your Clinical AI Isn’t Biased? California Splits the Duty

Your dispensing verification alert could be a liability. California may be poised to require proof that the clinical Artificial Intelligence (AI) tool informing patient care has been tested for biased impacts—and allocate that burden to both the developer and the provider using the tool.

This is the second installment in our series on California’s 2026 health care AI legislation. Each installment covers a single bill; this one addresses SB 503, which imposes health care-specific duties on developers and deployers concerning algorithmic bias. Organizations outside California should track this developer/deployer allocation closely, as it mirrors the structure other states have adopted in their emerging AI statutes.

This installment examines: how SB 503 divides bias identification, mitigation, and documentation duties between developers and deployers; what the required disclosures on training data and evaluation methodology mean for vendor diligence and contracting; and why organizations that build clinical AI in-house should expect to carry both roles at once.

In This Series

- Part One: Background and Prohibition on AI Independently Performing Licensed Clinical Functions (AB 1979)

- Part Two: Bias Identification and Mitigation for Clinical Decision Support Systems (SB 503)

- Part Three: Transparency and Disclosure Requirements for Clinical Decision Support Systems (AB 2575)

- Part Four: Regulation of AI in Psychotherapy Services (SB 903)

- Part Five: The Right to Human Customer Service Act (AB 1609)

- Series-Wide Action Items for Health Care Organizations

Part Two: Bias Identification and Mitigation for Clinical Decision Support Systems (SB 503)

What it does: If it becomes law, SB 503 would require developers and deployers of clinical decision support systems (CDSS) to identify systems with known or reasonably foreseeable risks of biased impacts in health programs or activities. The bill defines CDSS as an AI system that produces a prediction, classification, recommendation, evaluation, or analysis that aids clinical decision-making related to the timing of care, diagnosis, or treatment of patients. The definition explicitly excludes “appointment management, such as booking, canceling, and rescheduling appointments, appointment reminders, patient education and previsit materials and preparation, and payment processing.”

A biased impact, meanwhile, is broadly defined as an adverse impact on an individual based on their protected characteristics, including diminished access to health care, quality of care, or outcomes. The bill borrows the definition of protected characteristic used by the Unruh Civil Rights Act, which includes sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, and immigration status.

Key compliance obligations:

- Developers must make reasonable efforts to mitigate known or reasonably foreseeable risks of biased impacts and must provide deployers with: (1) a statement of intended uses and known risks; and (2) documentation covering training data demographics, evaluation methodology, bias mitigation efforts, and monitoring recommendations.

- Developers may comply by adhering to nationally recognized or widely adopted industry standards relevant to bias testing and AI in health care, or by providing results from regularly conducted algorithmic impact assessments.

- Deployers (i.e., providers) must regularly monitor CDSS and take reasonable and proportionate steps to mitigate known or foreseeable risks of biased impacts.

Effective date: Enrolled and presented to the Governor as of August 30, 2026; the standard effective date would be January 1, 2027, unless otherwise specified.

Analysis: SB 503 introduces a health care-specific algorithmic bias framework for granular developer/deployer transparency and bias-mitigation obligations that has no direct parallel in federal law.

Health care entities that both build and use CDSS in-house will bear dual developer/deployer responsibilities. The documentation requirements, including training data representativeness and bias evaluation methodology, will require close coordination between health IT, compliance, and clinical teams.

Pharmacy AI tools such as drug interaction checkers, automated dispensing verification systems, and clinical dosing recommendation engines qualify as CDSS if they produce predictions or recommendations aiding clinical decision-making related to treatment. Whether a pharmacy entity qualifies as a “deployer” depends on whether it is classified as a clinic, health facility, or group practice. As noted above, most stand-alone pharmacies (but not necessarily pharmacists) fall outside those categories.

The same analysis extends to:

- health insurers and managed care organizations, whose medical necessity, utilization management, and level-of-care models generate evaluations bearing directly on treatment;

- PBMs, whose prior authorization, step therapy, and specialty drug appropriateness engines perform that function at the benefit layer; and

- health technology vendors, device manufacturers, and analytics firms, whose predictive and diagnostic models are embedded in provider workflows.

However, pharmacies, payers, and health care adjacent entities may fall under the “developer” category if they develop proprietary clinical AI tools used in health care settings regardless of deployer status. Entities in that position should also expect to pass through SB 503’s documentation duties by contract, in the form of representations on training data demographics and evaluation methodology, delivery of intended-use and known-risk statements, and ongoing monitoring support, even where the statute itself does not classify them as deployers.

Finally, providers should also consider EHR integration points, where CDSS tools reach clinicians. EHR vendors that embed third-party CDSS modules may, themselves, qualify as developers under SB 503. Organizations should confirm—and document in contract—which entity (e.g., EHR vendor, CDSS developer, or provider) bears the documentation obligations at each integration point in the clinical workflow.

Practical takeaways:

- Inventory your CDSS and your role. Catalog AI systems that produce predictions, recommendations, or evaluations that have a bearing on diagnosis or treatment and determine whether the organization is acting as a developer, deployer, or both for each tool.

- Update vendor contracts. Build vendor diligence and contracting processes that require developer documentation on training data demographics, evaluation methodology, bias mitigation, and monitoring recommendations.

- Stand up a bias-monitoring workflow. Assign responsibility within existing AI governance structures (e.g., governance committee or clinical informatics review process) to regularly review CDSS outputs for biased impacts and to document mitigations.

- Establish your lodestar. Consider aligning bias testing with nationally recognized standards or recurring algorithmic impact assessments to establish a defensible compliance path.

In sum, SB 503 allocates bias identification and mitigation duties between those who build clinical decision support systems and those who deploy them. Part Three of this series discusses AB 2575, which builds on that allocation by requiring covered entities to inventory and disclose the very systems SB 503 obligates them to monitor.

For questions about how SB 503 applies to your organization or guidance on the development or deployment of AI in the health care setting, contact your Quarles attorney or:

- Meghan O’Connor: (414) 277-5423 / meghan.oconnor@quarles.com

- Candice Andalia: (202) 780-2627 / candice.andalia@quarles.com

Related Capabilities

- 340B Program

- Artificial Intelligence

- Data Asset Management, Privacy & Cybersecurity

- Health & Life Sciences

- Health Information Technology, Privacy and Security

- Health Insurance Industry Partners (PBMs, TPAs, DMPOs and URAs)

- Hospitals and Health Systems

- Long-Term Care and Senior Housing

- Pharmacy, Drug and Device

- Provider and Physician Groups

- Telehealth