Path From Predictive Analytics to Improved Patient Outcomes: A Framework to Guide Use, Implementation, and Evaluation of Accurate Surgical Predictive Models

    loading  Checking for direct PDF access through Ovid


An entire industry is booming on the promise that electronic health predictive analytics (e-HPA) can improve surgical outcomes by, for example, predicting whether a procedure is likely to benefit a patient compared with alternative treatments, or if a patient will experience short or long-term complications.1 The hope is that surgeons can use model predictions to improve the continuum of surgical care including patient selection, informed consent, shared decision making, preoperative risk modification, and perioperative management with the ultimate goal of producing better outcomes. Although so much effort is expended developing and selling predictive models, almost no attention is paid to rigorously testing if patients treated in health care settings that implement e-HPA have better outcomes than those treated in absence of e-HPA. Therefore, practicing surgeons need to be skeptical regarding the inherent benefits of e-HPA, as should model developers, implementers, and patients. Before buying or using an e-HPA system, all stakeholders should insist on research that rigorously evaluates not only the accuracy of the predictive models, but their effects on health outcomes when used in particular ways in real clinical settings. Without evidence of effectiveness and a careful examination of unintended consequences, implementation of e-HPA systems may produce more risks and/or costs than benefits.
Before describing the many ways even accurate and technically integrated e-HPA can fail to produce benefits to patients, we should review some characteristics of good predictive models, without which there is no hope. Good predictive models need to include important and modifiable outcomes, be adequately accurate (eg, sensitive, specific) given the clinical context, only include inputs that will be available at the time of decision or possible intervention, be cross-validated in data not used to develop the model, and be implemented in a usable and accessible technology platform. Moreover, model developers should be able and willing to transparently share the details of the model development methodology, model performance metrics, and model coefficients to facilitate replication, comparisons with alternative models, and refinements. The Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) standards and the recent Consensus Statement on E-HPA elaborate these points.2,3 Unfortunately, a disappointing number of publically available predictive models meet these criteria.4 However, developing good predictive models that meet these criteria is just the beginning of successful implementation and perhaps easier than the downstream challenges to which we now turn.
In rare cases when a particular implementation of e-HPA is evaluated and shown to produce real improvements in treatment quality and outcomes,5 it is generally under-appreciated how many intervening steps happened to translate model predictions into improved health. In prominent cases where the anticipated benefits of e-HPA failed to materialize,6 people are left wondering what went wrong along the pathway between accurate predictions and patient outcomes. Figure 1 represents a framework to better understand the conditions and events that must occur to enable good predictive models to create real value and benefit in the surgical context. The framework has a least 2 purposes. First, the framework is designed to sensitize all stakeholders, especially practicing surgeons, to the many implementation challenges that follow model development and technical integration. Second, the framework provides guidance to researchers and administrators who seek to rigorously test the clinical impacts of e-HPA—not only to estimate overall impacts, but to identify processes that might have broken down if no effects are observed.
Condition 1 is that model outputs need to be accessed by someone who has the potential to act in beneficial ways. Prediction models are not helpful if nobody knows about them.
    loading  Loading Related Articles