In a pioneering move to safeguard patient rights, California has enacted Senate Bill 1120, known as the “Physicians Make Decisions Act,” which took effect on January 1, 2025. This legislation restricts health insurers from using artificial intelligence (AI) and algorithmic tools to make autonomous decisions regarding the denial, delay, or modification of healthcare services based on medical necessity. Instead, such determinations must now be made exclusively by licensed healthcare professionals.

Background and Rationale

The increasing integration of AI in healthcare has streamlined various processes, including utilization management and claims processing. However, concerns have emerged regarding the potential for AI-driven systems to make erroneous or biased decisions, leading to unjust denials of necessary medical care. Instances have been reported where AI algorithms, lacking comprehensive clinical context, have overridden physicians’ recommendations, adversely affecting patient outcomes. A notable case involved allegations against a major insurer for using AI to automatically reject claims without proper individual assessments, highlighting the pressing need for regulatory oversight.

Key Provisions of SB 1120

SB 1120 introduces several critical requirements for health care service plans (HCSPs) and disability insurers operating in California:

• Human Oversight Mandate: Decisions concerning medical necessity must be made solely by licensed physicians or qualified healthcare professionals competent in evaluating the specific clinical issues involved. This ensures that patient care decisions are grounded in professional medical judgment rather than automated processes.
• Individualized Assessment Requirement: AI tools utilized in utilization reviews must base their analyses on an enrollee’s personal medical history, clinical circumstances, and relevant records. Reliance solely on generalized datasets is prohibited, promoting personalized patient evaluations.
• Transparency and Accountability: Insurers are obligated to disclose their use of AI tools in written policies and procedures. Additionally, these tools must be open to inspection for audits or compliance reviews, ensuring adherence to the stipulated guidelines.
• Non-Discrimination Clause: The legislation prohibits AI tools from discriminating, directly or indirectly, against enrollees in violation of federal or state laws. This provision addresses concerns about potential biases in AI algorithms that could lead to inequitable healthcare decisions.

Implications for Stakeholders

• Patients: The act enhances patient protection by ensuring that medical necessity decisions are made by qualified professionals who can consider the full scope of an individual’s health needs, thereby reducing the risk of inappropriate care denials.
• Healthcare Providers: Physicians and other healthcare professionals regain authoritative control over patient care decisions, fostering a healthcare environment where clinical expertise prevails over automated assessments.
• Insurers: While AI can still be employed to support decision-making processes, insurers must implement robust oversight mechanisms to ensure compliance with SB 1120. This may involve revising current practices, investing in training for healthcare professionals involved in utilization reviews, and ensuring transparency in AI tool usage.

Broader Context and Future Outlook

California’s legislative action reflects a growing national concern regarding the unchecked use of AI in critical decision-making areas, particularly in healthcare. Other states are observing these developments closely, and similar legislative initiatives may emerge nationwide. The federal Centers for Medicare and Medicaid Services (CMS) has also issued guidance emphasizing the necessity of human involvement in coverage decisions assisted by AI, aligning with the principles of SB 1120.

As AI technology continues to evolve, balancing innovation with ethical considerations remains paramount. California’s SB 1120 sets a precedent for ensuring that technological advancements in healthcare serve to enhance, rather than undermine, patient care and safety. Ongoing vigilance and adaptive regulatory frameworks will be essential to navigate the complexities introduced by AI in healthcare and other sectors.