Summary:
A recent study conducted at Yale highlights the limitations of current mathematical models in personalized medicine, particularly in the context of schizophrenia treatment. While these models can accurately predict patient outcomes within specific clinical trials, they struggle to maintain efficacy when applied to diverse trials. This discrepancy challenges the reliability of AI-driven algorithms in varied settings, emphasizing the importance of validating algorithms across multiple contexts to ensure their trustworthiness.
The study underscores the necessity for algorithms to demonstrate effectiveness in different scenarios before gaining widespread acceptance. It exposes a significant disparity between the potential of personalized medicine and its current practical implementation, especially considering the variability in clinical trials and real-world medical environments.
Key Points:
- Existing mathematical models in personalized medicine excel within specific clinical trials but lack generalizability across varying trial conditions.
- Concerns are raised regarding the application of AI and machine learning in personalized medicine, particularly in conditions like schizophrenia where individual treatment responses vary significantly.
- The research advocates for enhanced data sharing and the inclusion of additional environmental factors to enhance the reliability and precision of AI algorithms in medical applications.
Source: Yale
The pursuit of personalized medicine, a healthcare approach that tailors treatment based on a patient’s unique genetic makeup, has become a pivotal objective in the medical field. However, a recent study led by Yale reveals the limited effectiveness of current mathematical models in predicting treatments.
In an examination of multiple clinical trials focused on schizophrenia treatments, the researchers discovered that while the mathematical algorithms could forecast patient outcomes within the specific trials they were developed for, they faltered when applied to patients involved in different trials.
The findings were published on Jan. 11 in the journal Science.
Adam Chekroud, an adjunct assistant professor of psychiatry at Yale School of Medicine and the paper’s corresponding author, commented on the study’s implications, stating, “This study truly challenges the conventional approach to algorithm development and sets a higher standard for future endeavors. It is imperative for algorithms to prove their efficacy in at least two distinct settings before garnering substantial enthusiasm.”
Chekroud, who is also the president and co-founder of Spring Health, a mental health services company, emphasized the pressing need for more personalized treatments in conditions like schizophrenia. With approximately 1% of the U.S. population affected by schizophrenia, the disorder exemplifies the necessity for tailored therapies, given that up to 50% of diagnosed patients do not respond to initial antipsychotic drugs.
The potential of new technologies utilizing machine learning and artificial intelligence to develop algorithms capable of predicting treatment responses for individual patients offers hope for improved outcomes and reduced healthcare costs.
However, due to the high expenses associated with conducting clinical trials, most algorithms are solely tested within a single trial. Researchers initially believed that these algorithms would be effective when tested on patients with similar profiles and undergoing similar treatments.
To validate this assumption, Chekroud and his team aggregated data from five schizophrenia treatment clinical trials accessible through the Yale Open Data Access (YODA) Project, which advocates for responsible sharing of clinical research data.
While the algorithms consistently predicted patient outcomes within the trials they were designed for, they failed to do so for schizophrenia patients in other clinical trials.
Chekroud explained that the primary issue lies in the fact that most medical research algorithms are intended for use with more extensive datasets. Clinical trials, which are resource-intensive and time-consuming, typically enroll fewer than 1,000 patients.
Applying sophisticated AI tools to analyze these smaller datasets can lead to “over-fitting,” where a model learns response patterns specific to the initial trial data but lacks generalizability when new data are introduced.
Chekroud emphasized the necessity of developing algorithms akin to the process of creating new drugs, requiring validation across diverse times and contexts to establish credibility.
The researchers suggested that incorporating additional environmental variables might enhance algorithm success in analyzing clinical trial data. Factors such as substance abuse or social support can significantly influence treatment outcomes.
While clinical trials adhere to stringent criteria to enhance success rates, real-world medical settings present a broader patient spectrum and greater treatment variability, posing additional challenges for algorithm application.
John Krystal, the Robert L. McNeil, Jr. Professor of Translational Research and professor of psychiatry, neuroscience, and psychology at Yale School of Medicine, underscored the complexity of using algorithms in clinical practice if they cannot generalize across various trials.
Efforts to share data among researchers and accumulate additional data from large-scale healthcare providers could potentially enhance the accuracy and reliability of AI-driven algorithms, Chekroud proposed.
Though the study focused on schizophrenia trials, its implications extend to personalized medicine in broader contexts such as cardiovascular disease and cancer, according to Philip Corlett, an associate professor of psychiatry at Yale and a study co-author.
Other contributors to the study from Yale include Hieronimus Loho, Ralitza Gueorguieva, a senior research scientist at Yale School of Public Health, and Harlan M. Krumholz, the Harold H. Hines Jr. Professor of Medicine (Cardiology) at Yale.
About this AI and personalized medicine research news
Author: Bess Connolly
Source: Yale
Contact: Bess Connolly – Yale
Image: Image credited to Neuroscience News
Original Research: Closed access.
“Illusory generalizability of clinical prediction models” by Adam Chekroud et al. Science
Abstract
Illusory generalizability of clinical prediction models
The widespread anticipation that statistical models can enhance medical treatment decision-making is often based on observations of a model’s success in one or two datasets or clinical contexts due to the scarcity and cost of medical outcomes data.
We critically examined this optimism by evaluating the performance of a machine learning model across several independent clinical trials involving antipsychotic medication for schizophrenia.
While these models accurately predicted patient outcomes within the trial they were developed for, they performed no better than random chance when applied out-of-sample. Even pooling data from multiple trials did not enhance predictive accuracy.
These outcomes suggest that models predicting treatment outcomes in schizophrenia are highly context-specific and may lack generalizability.
