A statistical method called “Magnitude-Based Inference” (MBI) has gained a following in the sports science literature, despite concerns voiced by statisticians. Its proponents have claimed that MBI exhibits superior Type I and Type II error rates compared with standard null hypothesis testing for most cases. I have performed a re-analysis to evaluate this claim.Methods
Using simulation code provided by MBI’s proponents, I estimated Type I and Type II error rates for clinical and non-clinical MBI for a range of effect sizes, sample sizes, and smallest important effects. I plotted these results in a way that makes transparent the empirical behavior of MBI. I also re-ran the simulations after correcting mistakes in the definitions of Type I and Type II error provided by MBI’s proponents. Finally, I confirmed the findings mathematically; and I provide general equations for calculating MBI’s error rates without the need for simulation.Results
Contrary to what MBI’s proponents have claimed, MBI does not exhibit “superior” Type I and Type II error rates to standard null hypothesis testing. As expected, there is a tradeoff between Type I and Type II error. At precisely the small-to-moderate sample sizes that MBI’s proponents deem “optimal,” MBI reduces the Type II error rate at the cost of greatly inflating the Type I error rate—to two to six times that of standard hypothesis testing.Conclusions
MBI exhibits worrisome empirical behavior. In contrast to standard null hypothesis testing, which has predictable Type I error rates, the Type I error rates for MBI vary widely depending on the sample size and choice of smallest important effect, and are often unacceptably high. MBI should not be used.