MBI is a
Rigorous and Valuable Statistical Tool: a Comment on The Vindication of Magnitude-Based Inference Mick Wilkinson Sportscience
22, sportsci.org/2018/CommentsOnMBI/mw.htm, 2018 Summary: Recent criticisms
about error rates in MBI are based on flawed logic, false assertions, and
attempts to apply error-type definitions that do not and cannot apply to MBI.
MBI permits probabilistic statements about the magnitude of the influence of
a predictor on an outcome variable with practical/clinical context and
uncertainty about the magnitude. It has a robust and rigorous theoretical
basis and deserves to be recognised as a unique and valuable addition to a
researcher’s statistical toolbox. The recent publication of Sainani (2018), and the social media circus that followed, highlighted the generally poor understanding of both the philosophical intent and practical use of statistical inference for addressing questions characteristic of science in general and sport and exercise science in particular. Most questions in sport, exercise and medicine, especially those about interventions, concern the influence of x (some explanatory/independent variable) on y (some outcome/dependent variable). Whether x influences y (i.e., yes/no) is redundant as there will always be some influence even if it is vanishingly small. Moreover, with a continuous outcome variable (as is usually the case in sport and exercise science), the probability of no/zero influence is precisely zero. As such, null hypothesis significance testing (NHST) with a binary yes/no decision made against an arbitrary decision threshold (p <0.05) cannot address the question. Recent statements from the American Statistical Association and many published articles (including a recent short review by Wilkinson and Winter, 2018) are clear on these and related issues with NHST: the only worthwhile approaches to statistical inference are those based on estimating the magnitude of the influence of x on y. Estimation of effect magnitudes must also express the uncertainty around the estimate caused by noise, sample variation, etc. Confidence intervals have been proposed as a solution. To be of value, any statistical tool used must allow the user to say what they wish to say about their findings. In my experience and in reading how researchers tend to interpret confidence intervals, they wish to make probabilistic statements about the magnitude of the influence of x on y based on their observed data. However, there is a problem with confidence intervals. In the frequentist interpretation, the level of confidence (e.g., 95%) is not attached to the calculated confidence interval. Instead, 95% is the proportion of intervals that would contain the "true" effect magnitude, if the experiment were repeated an infinite number of times. As such, the confidence interval does not tell researchers anything with practical application, and it lacks context: an effect magnitude of practical or clinical importance against which the estimate of the true effect can be interpreted. A Bayesian approach, in which the confidence interval is interpreted as a "credibility" interval, is the practical alternative. Indeed, both recent critiques of MBI (Welsh and Knight, 2015; Sainani, 2018) recommended full Bayesian approaches as the way forward. While the Bayesian approach allows probabilistic statements to be attached to the calculated interval (there is such-and-such a chance that the true effect of x on y lies within the credibility interval), a fully Bayesian approach is informed by the researcher’s subjective prior belief expressed as a probability distribution. But now there is another problem: such a "prior" is rarely, if ever, justifiable. MBI offers an ideal solution. It provides an estimate of the true effect magnitude and its uncertainty in the form a Bayesian interval that is not affected by any prior belief: it uses a "flat" (non-informative) prior. Bayesian estimation with a flat prior is a well-established method. The unique addition of MBI is probabilistic interpretation of these intervals relative to a priori practically/clinically worthwhile effect magnitudes. In short, MBI provides the opportunity to make probabilistic statements about the magnitude of the influence of x on y with practical/clinical context. So why has it been so maligned? The motivation behind the recent attacks is less clear than their general thrust. Both critiques focus on errors in decisions that could be made based on the results of MBI analyses. No inferential method is error-free, as a conclusion is based only ever on one of the infinite number of possible samples. In their paper in Sports Medicine, Hopkins and Batterham (2016) accomplished something never before achieved: they presented definitions of Type-1 and Type-2 errors that permitted comparison between inferential methods that otherwise were impossible to compare. With these definitions, they used robust simulation to demonstrate generally lower error rates with MBI compared to NHST. Traditional definitions in NHST are the chances of wrongly rejecting or wrongly retaining a zero-effect null hypothesis in the long run (i.e., in the frequentist view of probability). Neither null hypotheses nor long-run error rates exist in MBI given it is a Bayesian-style estimation approach. Sainani’s criticism and claims about MBI error rates are based on erroneous definitions of error types not applicable to MBI and on misinterpretations of the definitions provided by Hopkins and Batterham. The failure of Sainani and others who support her case against MBI to recognise this inconsistency is worrying indeed. Thankfully, the full rebuttal to Sainani’s paper (Hopkins and Batterham, 2018) provides clear and detailed evidence of the false assertions and flaws in logic forming the basis of her criticisms. In summary, and in my experience, researchers wish to estimate the magnitude of the influence of x on y and make a probabilistic Bayesian-style statement about it based on their data. MBI permits such statements while expressing the uncertainty about the true effect and providing context for the effect relative to practically/clinically meaningful magnitudes. It has a robust and rigorous theoretical basis and deserves to be recognised as a unique and valuable addition to a researcher’s statistical toolbox. Hopkins WG, Batterham AM
(2018). The vindication of magnitude-based inference. Sportscience 22, 19-27 Wilkinson M, Winter EM
(2018). Estimation versus falsification approaches in sport and exercise
science. Journal of Sports Sciences 36, doi.org/10.1080/02640414.2018.1479116 Back to index of comments. Back
to The Vindication of
Magnitude-Based Inference. First published
17 June 2018. |