This is part of the NHST (Null Hypothesis Significance Testing) paradigm, also known as “tests”.
Details #
The principle of statistical significance significance tests is that the person doing the test defines in advance of doing the test the risk they are willing to accept of the test coming back as “significant” by mere sampling vagaries if there was no effect tested for at all in the population from which your data are a random sample. By historical convention this risk (the False Discovery Rate/Risk or FDR, or “alpha”) is almost always set at .05, or a one in twenty risk.
For one test that’s all clear, however, if you are doing multiple tests you run into the multiple tests issue/problem: if you set your alpha at .05 for each separate test your risk of finding one or even more spuriously significant test across the multiple tests you conducted goes up above .05 and rockets to very high probabilities if you have a fair few tests (see the multiple tests problem entry). To handle this one option is to set the individual test alpha values lower to keep the “family wise” (or “family-wise” or even “familywise”) false discovery rate at .05 (or whatever you want it to be).
Try also #
- Bonferroni correction
- False Discovery Rate (FDR)
- Holm-Bonferroni method
- Multiple tests problem
- Null hypothesis
- Null hypothesis significance testing (NHST) paradigm
- Population
- Sampling and sample frame
Chapters #
Not covered in the OMbook.
Online resources #
My Rblog entry on the Holm-Bonferroni method illustrates the issues of mainting the family-wise false discovery rate
Dates #
First created 20.ix.26, tweaked to add a link 21.ix.26.
