Understanding the Significance of Aerosolized Vasodilator Use in Pulmonary Hypertension: What Is Numerically, Statistically, and Clinically Meaningful?
An additional point that warrants consideration is that as multiple endpoints were being examined in comparisons being made to both placebo as well as to the aerosolized versions of the IV form of the vasodilators, the investigators appropriately planned an adjustment for the multiple comparisons to reduce the risk of false discovery. Accordingly, they reported a Bonferroni correction of 0.05/13 (ie, significance at P < .004) to account for the 13 hemodynamic variables that were evaluated. However, as they also examined 5 other nonhemodynamic variables (ie, inotrope use, vasopressor use, intensive care unit length of stay, hospital length of stay, and mortality), there were actually 18 different endpoints that needed adjusting for. Furthermore, as placebo versus vasodilator and IV versus aerosolized comparisons were examined, this Bonferroni correction could easily have been additionally adjusted by a multiple of 2—ie, with the total number of comparisons made actually then being 36 (ie, P = .05/36 = .0014). Although this would not change their stated “significance” for the 1 secondary endpoint they reported as such—ie, right ventricular ejection fraction (RVEF) was marginally improved by 7.29% (P < .0001)—it is nevertheless important that the appropriate level of adjustment be considered for all the other various comparisons.
Despite the stated corrected P value for (statistical) significance not being reached for any endpoint except for the RVEF, the authors state that “there was a significant reduction in PVR and significant increases in MAP” associated with the aerosolized agents. Though there were clear numerical decreases in pulmonary vascular resistance and accompanying increases in mean arterial pressure, the corresponding P values were well below the stated corrected liberal threshold of P = .004, and certainly well beneath the appropriately more conservative P = .0014 (as calculated above), indicating that they may not have been significant at all. Thus, it leaves the reader somewhat uncertain as to how confident one can be with what the “significant” effect of these aerosolized vasodilators actually is. Moreover, as the only endpoint (albeit secondary) reaching the liberal threshold for significance was RVEF—which itself was a variable only reported in 4 of the trials (total N = 123), including their own multicenter study2—one could easily question the clinical significance of the marginal increase that was reported. Additionally, as there was no clinically meaningful minimal effect specified a priori, the risk of over interpretation of this apparent change is quite possible.
Importantly, “significance” as a construct should really only be stated when there is a tacit acknowledgement of actual statistical significance (ie, the “statistical” being redundant and not needed as a preface) and should further only truly be considered meaningful if it meets a clinically significant threshold (which itself should ideally be defined a priori).