The Problem with “Magnitude-Based Inference”

    loading  Checking for direct PDF access through Ovid

Abstract

Purpose

A statistical method called “Magnitude-Based Inference” (MBI) has gained a following in the sports science literature, despite concerns voiced by statisticians. Its proponents have claimed that MBI exhibits superior Type I and Type II error rates compared with standard null hypothesis testing for most cases. I have performed a re-analysis to evaluate this claim.

Methods

Using simulation code provided by MBI’s proponents, I estimated Type I and Type II error rates for clinical and non-clinical MBI for a range of effect sizes, sample sizes, and smallest important effects. I plotted these results in a way that makes transparent the empirical behavior of MBI. I also re-ran the simulations after correcting mistakes in the definitions of Type I and Type II error provided by MBI’s proponents. Finally, I confirmed the findings mathematically; and I provide general equations for calculating MBI’s error rates without the need for simulation.

Results

Contrary to what MBI’s proponents have claimed, MBI does not exhibit “superior” Type I and Type II error rates to standard null hypothesis testing. As expected, there is a tradeoff between Type I and Type II error. At precisely the small-to-moderate sample sizes that MBI’s proponents deem “optimal,” MBI reduces the Type II error rate at the cost of greatly inflating the Type I error rate—to two to six times that of standard hypothesis testing.

Conclusions

MBI exhibits worrisome empirical behavior. In contrast to standard null hypothesis testing, which has predictable Type I error rates, the Type I error rates for MBI vary widely depending on the sample size and choice of smallest important effect, and are often unacceptably high. MBI should not be used.

Related Topics

    loading  Loading Related Articles