As artificial intelligence continues its rapid integration into various sectors, the healthcare industry is at a critical juncture, with major health systems beginning to deploy their own branded AI chatbots. This burgeoning trend aims to capitalize on the public’s growing reliance on large language models (LLMs) for health information, seeking to guide patients towards their services and offer a potentially safer alternative to commercial AI tools. However, this ambitious move is simultaneously igniting a flurry of questions and concerns within a healthcare system already grappling with significant challenges.
The driving force behind these initiatives is the undeniable reality that Americans are increasingly turning to AI for health advice. A recent KFF poll revealed that one in three adults have already consulted an AI chatbot for health-related queries, a statistic that highlights a significant shift in how individuals seek and consume health information. This widespread adoption is fueled, in part, by the systemic issues plaguing the U.S. healthcare landscape. For many, AI chatbots represent a readily accessible, albeit unvetted, resource in the face of prohibitive costs, a lack of consistent access to primary care providers—with nearly a third of Americans lacking one—and the sheer inconvenience of navigating a complex and often unresponsive system. When asked about their primary motivations for using AI for health, 19% cited an inability to afford care, and 18% pointed to the absence of a regular healthcare provider or difficulty securing appointments. A majority, 65%, simply sought a quick answer.
Health system executives champion these new AI offerings as a means to enhance patient convenience and promote digital equity, asserting that they are meeting patients "where they are." The promise is that these branded chatbots will provide a more secure and integrated experience, connecting directly to patient portals and electronic health records, thereby steering users toward the health system’s established network of care.
"We are at an inflection point in healthcare," stated Allon Bloch, CEO of clinical AI company K Health. "Demand is accelerating, and patients are already using AI to navigate their lives." K Health, in partnership with Hartford HealthCare, is at the forefront of this movement, rolling out its "PatientGPT" chatbot to tens of thousands of existing patients. Bloch further emphasized the imperative of developing these tools responsibly, stating, "The question isn’t whether AI will shape healthcare, it’s about how we do it in a safe, transparent way, inside a health system that connects to your medical records and your care team. PatientGPT represents that turning point."
Despite these optimistic projections, a significant segment of the healthcare community remains cautious. Experts are raising critical questions about the readiness of these AI chatbots for such high-stakes deployments, the adequacy of monitoring mechanisms, the complex issue of liability, and whether these tools truly address the fundamental care deficits that drive patients to seek AI assistance in the first place.
The Context of a Strained Healthcare System
To fully grasp the implications of AI’s entry into healthcare, it is crucial to examine the broader context of the U.S. healthcare system. Despite being one of the wealthiest nations globally, the United States consistently underperforms compared to other high-income countries in key health metrics. Americans experience lower life expectancy, a higher burden of avoidable deaths, and disproportionately high rates of maternal and infant mortality. Furthermore, the nation struggles with elevated rates of obesity and chronic conditions, coupled with a persistent lack of access to care and poorer overall health outcomes. The absence of universal healthcare coverage makes the U.S. a global outlier, contributing to a situation where millions are left without adequate medical support.
The current landscape, characterized by these systemic weaknesses, creates a fertile ground for the adoption of readily available AI tools. The accessibility and apparent user-friendliness of LLMs have positioned them as a first-line resource for many, even for sensitive health inquiries.
Mounting Concerns Over Accuracy and Misinformation
While the integration of AI into health systems is presented as a solution, the inherent risks associated with LLMs remain a significant concern. Recent studies have underscored the potential for inaccuracies and the ease with which misinformation can be propagated.
A study published in Nature Medicine in February, which involved nearly 1,300 participants, evaluated the medical accuracy of prominent LLMs, including GPT-4o, Llama 3, and Command R+. When researchers presented the AI models with specific medical scenarios in a controlled text format, they demonstrated a high degree of accuracy, correctly identifying medical conditions approximately 95% of the time and suggesting appropriate next steps (such as seeking emergency care) about 56% of the time. However, a stark contrast emerged when participants used their own natural language prompts to describe the same scenarios. In these real-world interactions, the LLMs could only correctly identify a medical condition about a third of the time and guided users to the appropriate next step in just 43% of cases.
"The study essentially shows that ‘people don’t know what they are supposed to be telling the model’," explained lead author Andrew Bean, an AI researcher at Oxford University. This disparity highlights a critical gap between the AI’s performance in curated environments and its reliability when faced with the complexities of human input. Senior author Adam Mahdi added, "The disconnect between benchmark scores and real-world performance should be a wake-up call for AI developers and regulators."
Adding to these concerns is the unsettling susceptibility of LLMs to fabricated information. In a recent incident, Nature News reported that LLMs were engaging users in discussions about "bixonimania," a skin condition entirely invented by researchers in Sweden. The researchers had intentionally published two fake studies online to test the AI’s ability to absorb and disseminate misinformation. The ease with which this fabricated condition was integrated into AI responses served as a stark warning about the potential for widespread dissemination of medical inaccuracies.
Health Systems Forge Ahead with Chatbot Deployments
Despite these well-documented risks, several healthcare systems are moving forward with their own AI chatbot initiatives. Hartford HealthCare and K Health’s PatientGPT, after a beta launch to select patients last month, is set to expand its rollout to tens of thousands more individuals this week.
Hartford HealthCare published a pre-print study (not yet peer-reviewed) detailing its "red teaming" approach—a rigorous iterative stress-testing process—to improve the chatbot’s performance, particularly in high-risk scenarios. The study suggested that this testing reduced the failure rate in high-risk situations from 30% to 8.5% over time. However, the real-world implications of this remaining failure rate and its potential severity remain unclear.
PatientGPT operates in two primary modes. The first is a general medical question-and-answer function that can incorporate patient-specific information. The second is a "medical intake" mode, where the chatbot guides patients through a series of clinical flowcharts based on their reported symptoms. Once sufficient information is gathered, the AI suggests a next step, which could range from scheduling a primary care appointment to seeking urgent or emergency care. If the latter is recommended, the chatbot ceases further interaction.
Hartford HealthCare has committed to ongoing monitoring of PatientGPT’s performance. During the pilot phase, every patient interaction was reviewed by humans. Post-expansion, the system will shift to human review of 20 interactions daily, with a separate AI agent monitoring the remainder. Additionally, batch studies will be conducted on every 1,000 conversations.
Jeff Flaks, president and CEO of Hartford HealthCare, articulated the system’s patient-centric vision: "We’re on a mission to be the most consumer-centric health system in the country. So much of healthcare has traditionally been organized around the provider, but it’s clear we have to meet people where they are and where they desire to be met. With PatientGPT we are introducing a new tool that supports your health and provides access to a 24/7 care team, while protecting the human relationships at the heart of care."
Epic’s Emmie: A More Cautious Approach
Beyond PatientGPT, another significant player in the AI healthcare space is Emmie, an AI chat assistant developed by Epic, the dominant electronic health records provider for many U.S. hospitals. Several health systems, including Sutter Health in California and Reid Health in Indiana, are gradually integrating Emmie into their patient portals.
Judy Faulkner, Epic’s founder and CEO, described Emmie as a tool designed to assist patients in preparing for appointments by drafting visit agendas and, subsequently, helping them understand test results and address follow-up questions. While Emmie aims to enhance patient engagement, its functionality is deliberately constrained.
Sutter Health’s frequently asked questions regarding Emmie emphasize its limitations: the chatbot can "answer general health questions, and find or summarize information already visible in your chart—such as notes, results, past visits or messages." Crucially, it states that Emmie "doesn’t give personalized medical advice or make care decisions. Emmie is not intended for use in the diagnosis of disease or other conditions, or in the cure, mitigation, treatment or prevention of disease. Emmie is also not intended to replace, modify or be substituted for a physician’s professional clinical judgment."
Currently, Emmie is accessible to a limited subset of Sutter patients, who are encouraged to provide feedback on its responses through simple thumbs-up or thumbs-down ratings.
Reid Health, serving largely rural communities, is the second health system to adopt Emmie. Muhammad Siddiqui, CIO at Reid Health, explained the system’s rationale: "Patients want clearer answers, easier access and more guidance between visits. If we can provide that inside the health system experience, in a way that is connected to trusted clinical workflows, that is a much better path than leaving people on their own with public tools that may or may not be accurate." This perspective underscores the desire of health systems to offer a controlled and integrated AI experience, thereby mitigating the risks associated with patients relying on unverified external AI tools.
Implications for the Future of Healthcare
The rollout of these health system-specific AI chatbots marks a pivotal moment, representing a proactive attempt by the industry to harness a powerful emerging technology while attempting to mitigate its inherent risks. The dual motivations are clear: to enhance patient engagement and streamline access to care, while simultaneously attempting to capture and retain patients within their respective ecosystems.
The long-term implications of this trend are multifaceted. On one hand, if these chatbots can be reliably developed and rigorously monitored, they could indeed improve patient access to information and administrative support, particularly for those who face barriers to traditional care. The integration with electronic health records offers a significant advantage over standalone commercial AI tools, promising a more personalized and contextually relevant experience.
However, the fundamental challenges of AI accuracy, data privacy, and the potential for exacerbating existing health inequities cannot be overlooked. The success of these initiatives will hinge on the ability of health systems to demonstrate not just technological advancement, but a genuine improvement in patient outcomes and a commitment to transparency and ethical AI deployment. The critical question remains: can these branded chatbots effectively bridge the gap in care, or will they become another layer of complexity in an already overburdened system, potentially introducing new risks alongside their purported benefits? The coming years will be crucial in determining whether this technological leap forward leads to a more accessible and effective healthcare future or simply amplifies the existing challenges.
