Does Participation in the ACS-NSQIP Improve Outcomes?
In their recent analysis of the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP), Cohen et al1 conclude that participation in the ACS-NSQIP is associated with improved outcomes over time. We have significant concerns that these data should not be used to infer a causal relationship between participation in the ACS-NSQIP and improved outcomes over time, as their analysis did not include a comparison group and, therefore, did not control for secular trends over time.
Cohen et al1 present data from hospitals participating in the ACS-NSQIP between 2006 and 2013. In their analysis, they rely on hierarchical models to show a statistically significant improvement in outcomes of mortality, morbidity, and SSI within these hospitals. There is, however, the very real possibility that similar improvements occurred during the same period within hospitals that were not participating in the ACS-NSQIP.
An analysis of data within the article by Cohen et al1 illustrates this possibility. Cohen et al, in Table 1 of their article, report annual mortality and morbidity rates of hospitals, stratified by year of entry into the ACS-NSQIP (as shown in their Table 1). Shown below in Figure 1 are the adverse event rates for hospitals during the first year of their participation in ACS-NSQIP. This figure demonstrates improvements over time, even in hospitals that had not previously participated in the ACS-NSQIP.
To draw appropriate inferences regarding the association between hospital-based ACS-NSQIP participation and improvements in surgical outcomes, consideration has to be made for improvements over time within the universe of non-ACS-NSQIP hospitals. Two recent articles conducted this type of analysis (difference-in-differences) on the basis of data within 2 distinct sources—Medicare and the University HealthSystem Consortium.2,3 In both of these analyses, there was no association between hospital-based participation in the ACS-NSQIP and improvements in postoperative outcomes over time.
The authors do acknowledge this flaw in their study design, stating that a lack of control group exists with equivalent robust data. Despite this acknowledgment, the authors continue to state that their study (and previous studies) leads them to believe that “participation in ACS-NSQIP improves quality to a greater extent than secular trend (or is a critical component of observed trends).” They proceed to estimate that a large hospital participating in the ACS-NSQIP will avoid as many as 14 deaths and 300 complications over 5 years of participation. The implication is that non-ACS-NSQIP hospitals will not observe these same improvements. However, without a comparison group, it seems impossible to estimate the number of lives saved and complications avoided from participation alone.
The ACS-NSQIP should be applauded for continuing to encourage the measurement of surgical outcomes. Participation in a quality measurement platform is, however, only 1 part of a comprehensive approach to quality improvement. For real quality improvement to take place, outcomes data need to be used to guide changes in practice. Across the country, hospitals are engaged in a broad range of quality improvement efforts, and as a result, rates of adverse surgical outcomes are improving steadily over time. We caution against a conclusion that participation in the ACS-NSQIP is directly responsible for a degree of improvement in outcomes that is greater than this national trend.