Path From Predictive Analytics to Improved Patient Outcomes: A Framework to Guide Use, Implementation, and Evaluation of Accurate Surgical Predictive Models
Before describing the many ways even accurate and technically integrated e-HPA can fail to produce benefits to patients, we should review some characteristics of good predictive models, without which there is no hope. Good predictive models need to include important and modifiable outcomes, be adequately accurate (eg, sensitive, specific) given the clinical context, only include inputs that will be available at the time of decision or possible intervention, be cross-validated in data not used to develop the model, and be implemented in a usable and accessible technology platform. Moreover, model developers should be able and willing to transparently share the details of the model development methodology, model performance metrics, and model coefficients to facilitate replication, comparisons with alternative models, and refinements. The Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) standards and the recent Consensus Statement on E-HPA elaborate these points.2,3 Unfortunately, a disappointing number of publically available predictive models meet these criteria.4 However, developing good predictive models that meet these criteria is just the beginning of successful implementation and perhaps easier than the downstream challenges to which we now turn.
In rare cases when a particular implementation of e-HPA is evaluated and shown to produce real improvements in treatment quality and outcomes,5 it is generally under-appreciated how many intervening steps happened to translate model predictions into improved health. In prominent cases where the anticipated benefits of e-HPA failed to materialize,6 people are left wondering what went wrong along the pathway between accurate predictions and patient outcomes. Figure 1 represents a framework to better understand the conditions and events that must occur to enable good predictive models to create real value and benefit in the surgical context. The framework has a least 2 purposes. First, the framework is designed to sensitize all stakeholders, especially practicing surgeons, to the many implementation challenges that follow model development and technical integration. Second, the framework provides guidance to researchers and administrators who seek to rigorously test the clinical impacts of e-HPA—not only to estimate overall impacts, but to identify processes that might have broken down if no effects are observed.
Condition 1 is that model outputs need to be accessed by someone who has the potential to act in beneficial ways. Prediction models are not helpful if nobody knows about them.