Also known as the Holm-Bonferroni correction or just the Holm method/correction. This is an improvement on the Bonferroni method of mitigating the loss of statistical power for any one test when correcting for the multiple tests issue/problem to keep the family-wise False Discovery Rate (FDR) at whatever you want, typically .05.
That’s quite a melange of terms each has its own glossary entry, see below.
Details #
Unlike the simple Bonferroni method, which uses the same alpha for each individual test in the family of tests, usually .05/n where n is the number of tests, the Holm-Bonferroni method is sequential working on the tests in the order of their increasing p value in your data. This means that, if first test, i.e. the smallest p value, came out under .05/n then later will be against a slightly less stringent alpha value than .05/n and hence have slightly more statistical power than they would have had using the simple Bonferroni method.
For a worked example, go to my post in my Rblog that shows how this works: here.
While it’s neat maths and statistical theory and does what it sets out to do, nevertheless, having played around creating that Rblog entry, it seems to me that the gains in statistical power can be tiny or zero or quite large in a way that is very, very dependent on the spread of the p values in your analyses.
More generally this can become a way of making rather unthinking obedience to the Null Hypothesis Significance Testing (NHST) paradigm look more impressive because it invoked this posh sounding (and neat!) method. I suspect that in our field far more will be added to our understanding of implications of an analysis of the dataset with its multiple tests if the issues being explored, and the limitations of the NHST, are taken seriously in the discussion section of the paper. Sometimes that will immediately show that agonising about the multiple tests “problem” is meaningless as the global null model was uninteresting as it was never going to fit the data, that’s typically true for tests of the correlation coefficients between items in even a pretty badly formulated questionnaire!
Try also #
- Bonferroni method
- False Discovery Rate (FDR)
- Multiple tests issue/problem
- Null Hypothesis Significance Testing (NHST)
Chapters #
Not covered in the OMbook.
Online resources #
My Rblog post illustrating the method: here.
Dates #
First created 21.ix.26.
