By Paul Cerrato
”The complexity of medicine now exceeds the capacity of the human mind.” It’s still one of the best quotes I’ve seen that explains why physicians and nurses need help making wise clinical decisions. There is just too much information for any one clinician to hold within memory and remain effective. About 5,000 medical journal articles are published every day, making it impossible for the average clinician to keep up. Similarly, the electronic health record for many patients contains pages and pages of information. No one can reasonably be expected to wade through all this. Clinical decision support systems can help address this avalanche of data but even these systems often fall short.
In our new book, Reinventing Clinical Decision Support, John Halamka, MD, the President of the Mayo Clinic Platform, and I dive into some of the innovative ways to make these digital tools smarter and more sophisticated with the help of AI, machine learning, and advanced data analytics. John and I had planned to give two presentations at HIMSS 2020 but as most readers know, the conference was cancelled. With that in mind, here’s a brief summary of what I was going to talk about.
Do We Need Better Clinical Support Tools?
The graphic below illustrates some of the problems that exist in CDS systems. Typically, these systems rely on a knowledge base consisting of large clinical trials, clinical guidelines from medical specialty associations, and the like. But when you take a closer look at many of these trials over the years, numerous problems emerge. In 1987, for instance, a review in the New England Journal of Medicine uncovered 71 randomized controlled trials (RCTs) that rejected potentially valuable treatments. The trials had jumped to the conclusion that the treatments were useless because the studies had not included enough patients. These false negative studies all committed a Type 2 statistical error. In 1994, the track record hadn’t improved. This time, a JAMA review found that more than 300 RCTs claimed a treatment approach was useless—without sufficient evidence to support those claims.
Machine Learning-based Algorithms Can Improve CDS
Fast forwarding to 2013, there’s still reason to believe that many clinical trials fall short—but for a different reason. A case in point is the Look AHEAD study.(1) This large scale RCT assigned over 5,000 overweight and obese patients with type 2 diabetes to either an intensive lifestyle modification program or to a control group that only received supportive education. The investigators’ goal was to determine if the lifestyle program would reduce the incidence of death from cardiovascular disease, non-fatal myocardial infarction, non-fatal stroke, or hospitalization for angina. The original plan was to follow these patients for as long as 13.5 years, but the study was terminated early because there were no significant differences between the intervention and control groups. The lower caloric content and increased exercise in the intensive lifestyle group did have a positive impact, helping patients to lose weight, but it did not reduce the rate of cardiovascular events. (2)
The Look AHEAD researchers performed a subgroup analysis to see if certain portions of the patient population may have experienced better clinical outcomes. They looked more closely at patients by gender, race, or ethnic group, and the presence or absence of cardiovascular disease when they enrolled in the study. A forest plot analysis found no significant differences.
Aaron Baum and his colleagues at Icahn School of Medicine at Mount Sinai, New York, and elsewhere were not satisfied with those results and decided to do a much more in-depth subgroup analysis.(3) They used an advanced ML method called random forest modeling. One advantage of this method is that it does not start with a specific hypothesis on what variables or patient population characteristics may be responsible for possible heterogenous treatment effects. The original research project, on the other hand, hypothesized that a patient’s
gender, ethnicity, or history of cardiovascular disease may have influenced their response to treatment. During random forest analysis, a series of decision trees are created—thus the term “forest.” Initially, the technique randomly splits all the available data—in this case, the stored characteristics of about 5,000 patients in the Look AHEAD study—into two halves. The first half serves as a training data set to generate hypotheses and construct the decision trees. The
second half of the data serves as the testing data set. As Baum et al. explain, “The method first identifies subgroups with similar treatment effects in the training data, then tests the most promising heterogeneous treatment effect (HTE) hypotheses on the testing data . . .” (3)
Using this technique, Baum et al. constructed a forest that contained 1,000 decision trees and looked at 84 covariates that may have been influencing patients’ response or lack of response to the intensive lifestyle modification program. These variables included a family history of diabetes, muscle cramps in legs and feet, a history of emphysema, kidney disease, amputation, dry skin, loud snoring, marital status, social functioning, hemoglobin A1c, self-reported
health, and numerous other characteristics that researchers rarely if ever consider
when doing a subgroup analysis. The random forest analysis also allowed the investigators to look at how numerous variables interact in multiple combinations to impact clinical outcomes. The Look AHEAD subgroup analyses looked at only 3 possible variables and only one at a time.
In the final analysis, Baum et al. discovered that intensive lifestyle modification averted cardiovascular events for two subgroups, patients with HbA1c 6.8% or higher (poorly managed diabetes), and patients with well-controlled diabetes (Hba1c < 6.8%) and good self-reported health. That finding applied to 85% of the entire patient population studied. On the other hand, the remaining 15% who had well controlled diabetes but poor self-reported general health
responded negatively to the lifestyle modification regimen. The negative and positive responders cancelled each other out in the initial statistical analysis, falsely concluding that lifestyle modification was useless. The Baum et al. reanalysis lends further support to the belief that a one-size-fits-all approach to medicine is inadequate to address all the individualistic responses that patients have to treatment.
For more details on how machine learning was used in this study, see the video below.
This data analytics approach has been used to reevaluate several clinical trials and to demonstrate that reinventing clinical decision support requires the reinvention of data analytics.
References
- The Look AHEAD Research Group. (2013). “Cardiovascular Effects of Intensive
Lifestyle Intervention in Type 2 diabetes.” New England Journal of Medicine,
vol. 369, pp. 145–154.
- Cerrato, P, Halamka J. Reinventing Clinical Decision Support: Data Analytics, Artificial Intelligence, and Diagnostic Reasoning. Taylor and Francis/CRC Press, 2020. https://www.routledge.com/Reinventing-Clinical-Decision-Support-Data-Analytics-Artificial-Intelligence/Cerrato-Halamka/p/book/9780367186234
- Baum, A., Scarpa, J., Bruzelius, E., Tamler, R., Basu, S., and Faghmous, J. (2017).
“Targeting Weight Loss Interventions to Reduce Cardiovascular Complications of
Type 2 Diabetes: A Machine Learning-Based Post-Hoc Analysis of Heterogeneous
Treatment Effects in the Look AHEAD Trial.” Lancet Diabetes 7 Endocrinology,
vol. 5, pp. 808–815.
Recent Comments