All too often, particularly in our field, statistical reports are treated as definitive answers and the realities are hidden. These realities are always that statistical analyses look at data in the light of a model and that model has assumptions. It’s pretty tragic that statistical analyses are so often reported as if the assumptions don’t matter and don’t introduce uncertainties that should be considered when weighing up what the analyses tell us.
However, the anti-quantitative warriors sometimes suggest that as there are pretty much always assumptions that will be violated we should simply abandon statistical analyses. (OK, it’s never said that bluntly.) That’s like saying that because speedometers in cars are never perfectly accurate so we should stop fitting them to cars.
Perhaps just as unhelpful is the tendency to seem to blame data for not fitting the statistical assumptions and then to report things as if therefore the findings tell us nothing. (That also comes not from the authors but from critics of course.)
So we need our reports to be clear about what statistical models were pertinent, what their assumptions were, and how much it might matter that the data essentially never exactly fit the assumptions.
Details #
Statistical assumptions vary across the models and analyses used but typical ones are:
- That the data are random samples from infinitely large populations. There are statistical methods that don’t require infinite populations but I don’t think I’ve ever seen them used in our field so this assumption is essentially never satisfied by our data. Whether that matters is complex and rarely considered but if your dataset sizes are over 50 this is rarely, in my view, a serious problem.
- That, except for the effects in the model, e.g. the values of each data point is independent of those of any other data point. What does that mean? That if you are looking at whether there are differences in change trajectories of any individual’s measure scores per session vary across a collection of therapists then clearly the data within any therapist are not independent of each other: they share the therapist. However, that’s part of the model. Similarly, trajectories imply that values on a later session are not independent of any earlier value, they are likely to be correlated. However, again as we are looking at trajectories, that’s again part of the model. But if the therapies are group therapies and if we think that what is happening for each member of the group probably affects change for at least some other members then that is violation of independence of observations that either has to built into the model (difficult for realistic group sizes) or else it is a known violation of this assumption. This is one of the issues that can have huge impacts on findings.
- The other key assumption in statistical models are ones about the distributions of the data. These can apply to count data and to continuous variables.
- For count data this is about the processes that create the “yes/no” issues being counted. This is getting to the edge of my understanding but I can understand enough to know that if the model has this wrong then analyses can be misleading.
- The assumptions about distributions of continuous scores are more often important and the classical models were almost all based on the assumption that the distributions of values were Gaussian (“Normal”). Gaussian distributions are symmetrical, have a single mode (most common value) and the scores range from minus infinity to plus infinity. Although there are many good reasons why the Gaussian model dominated for the better part of a century and reasons why distributions of some continuous variables would be close to Gaussian, or could be transformed to near Gaussian, the last 40 to 50 years have seen huge changes in distributional assumptions built into statistical methods mostly based on the use of “bootstrapping” or other “machine intensive”/”computationally intensive” methods. Having said that, this takes us into some seriously complex issues and those methods are not the panaceas that they are sometimes taken to be. (I’d admit to having underestimated the issues at times until about 2020!)
There are not magic answers to these issues because statistical methods can be wonderful but they aren’t magic! To me Wagenmakers, et al. 2021. ‘Seven Steps toward More Transparency in Statistical Practice’. Nature Human Behaviour 5(11):1473–80. doi:10.1038/s41562-021-01211-8. Is a bit of counsel of perfection and I’ve never had the resources to achieve all that they recommend but their points are excellent:
Here, to promote such a practice, we recommend seven concrete
statistical procedures: (1) visualizing data; (2) quantifying inferential uncertainty; (3) assessing data preprocessing choices;
(4) reporting multiple models; (5) involving multiple analysts; (6) interpreting results modestly; and (7) sharing data and code.
We discuss their benefits and limitations, and provide guidelines for adoption. Each of the seven procedures finds inspiration in
Merton’s ethos of science as reflected in the norms of communalism, universalism, disinterestedness and organized scepticism.
We believe that these ethical considerations—as well as their statistical consequences—establish common ground among data
analysts, despite continuing disagreements about the foundations of statistical inference.
This issue of modesty, the literature also writes about “humility”, is crucial. We need to drop the idolising of statistical methods but I don’t hold out much hope for this happening with the pressures on modern researchers in its industrial model!
Try also #
- Bootstrap methods
- Computer intensive statistical methods
- Gaussian distribution
- Independence of observations
Chapters #
This is not covered very explicitly in the OMbook but the underlying principles are there throughout the book
Online resources #
I’d love to do some simulations demonstrating the issues but don’t hold your breath!
Dates #
First created 5.iii.26.
