This is a key part of the NHST (Null Hypothesis Significance Testing) paradigm, see null hypothesis and inferential testing for more on that generally. Within that paradigm the type I error is the risk you run of rejecting the null hypothesis if it is true for the population. That is to say the risk of declaring your finding “non-significant” even though, in the population from which you have your sample, the null hypothesis is not true and actually there is something systematic going on.
This can sound hard to follow and a bit backwards, certainly I found it so at first, however I came to understand its elegance and firm logic. Decades ago, teaching statistics to trainee psychiatrists we were in an era of rude stickers in the back of cars saying things like “surfers do it standing up” (with silhouette images to explain the allusion) and I used to say that hypothesis testing merited the slogan “statisticians do it backwards, but beautifully|logically|coherently. Maybe that still helps.
The risk of making such a type I error is something you (in principle) set before you start doing your study (see “pre hoc hypothesis” and “pre hoc hypothesis”). By a rather intriguing and highly debatable historical convention this is usually .05, i.e. that you are accepting a one in twenty risk of a “false positive”, of rejecting the null and accepting the alternative that something systematic of potential interest (in the terms of the test) is happening, when in fact in the population there is no effect.
All this is fine but rests on our data collection fitting the model of purely random sampling from infinitely large populations which are never true in our fields and the dichotomisation “significant” versus “non-significant” became a very unhelpful fetish in many research areas, not just ours. The paradigm is being re-evaluated by the statistical world thank goodness, and estimation and confidence intervals gaining the ground against the NHST they should always have had, but that’s beyond this glossary entry.
Details #
Nothing more here yet!
Try also #
Confidence Intervals (CIs)
Estimation
Inferential testing, “tests”
Null hypothesis
Null Hypothesis Significance Testing (NHST) paradigm
Pre hoc hypothesis
Post hoc hypothesizing
Chapters #
Chapter 5
Online resources #
Nothing yet, will try to set up a shiny simulation.
Dates #
First created 20.viii.23, tweaked and improved with active links 12.x.26.
