Oct. 23, 2026
Last time, I gave a fictional (though in one sense very real) example of a person who has demonstrated learning/understanding of statistics in each level of Bloom’s Taxonomy, despite not truly understanding the subject in much depth.
In one sense, this person is certainly accomplished, in that they have completed all the steps necessary to be awarded a graduate degree. It probably even took a lot of hard work and energy over multiple years.
But if the entirety of someone’s knowledge in a subject boils down to key terms, written procedures, and decision rules, it’s debatable whether we could really say they “understand” what’s going on.
It would be like charting the surface area of a lake but never looking underwater. The vast majority of the lake’s characteristics exist below the surface.
(By the way, this kind of education really does happen, and sometimes such individuals go on to become statistics educators! And then the cycle continues.)
What’s the problem with flowcharts and decision rules?
There’s nothing inherently wrong with them. But the fundamental reason that statistics educators teach them is to avoid having to explain what’s really going on.
Two possible reasons for this include:
- Statistics educators themselves don’t understand beyond the flowcharts and decision rules.
- Statistics educators do understand more deeply, but they don’t believe their students are capable of deeper understanding.
For example, consider the hotly debated p-value…
Many instructors simplify the concept to something like, “If this value is less than .05, then the research hypothesis is supported” (a semi-arbitrary decision rule).
If they’re teaching the concept like this, it’s also highly unlikely they’re explaining how to calculate p-values; more likely, they’re teaching how to click buttons to make the p-value appear.
So the learning ultimately boils down to instructing the computer to do something for you, then making conclusions about the nature of reality based on what shows up on the screen.
I knew one stats instructor who actually said, “If you see p < .05, then it’s time to celebrate! We got what we wanted!”
This reflects such a profoundly confused understanding of not only statistics but science more broadly that I’m almost at a loss for words. Almost.
So why do many educators avoid giving the true explanation for p-values?
Because the true answer is much harder to explain and understand!
The simplest accurate version would be something like, “Imagine the focal effect is literally zero*. If we ran this exact study an infinite number of times and plotted all the resulting test statistics, they would pile up like this [indicate a plot with the appropriate null sampling distribution]. Then we find where our actual observed test statistic lies in this distribution. The p-value is the proportion of area under this curve which is at least as extreme as our observed statistic. So it represents the probability we’d see a result at least as extreme as ours if the true effect were literally zero. This gives us some sense of how plausible the null hypothesis is.”
(*Note that this explanation assumes the null hypothesis is the nil hypothesis [i.e., the effect is 0]. We could actually use any null we wanted in a significance testing framework like this.)
I can assure you from experience that even highly intelligent people need that concept explained to them repeatedly and with multiple visuals before they start to develop some intuition for it.
But once you do understand, then you almost become unable to interpret the result according to an arbitrary decision rule like, “If p < .05, reject the null hypothesis.”
Someone who genuinely understands can only see a p-value for what it truly is: A piece of information about what the world looks like if X is true, which therefore also serves as evidence regarding the plausibility of X.
Decision rules like α = .05 exist for people who don’t understand the underlying concepts well enough to make their own judgments.
Flowcharts and decision rules exist to help us avoid understanding.
