Oct. 26, 2026
In Part 1, I compared shallow and deep understanding Levels 1 and 2 of Bloom’s Taxonomy (Knowledge and Comprehension, respectively, to use the original terminology).
In Part 2, I did the same for Levels 3 and 4 (Application and Analysis).
In this final installment, I do the same for Levels 5 and 6 (Evaluation and Synthesis).
Note that the names of Bloom’s levels, as well as the order, are different depending on which version you’re looking at, but it doesn’t really matter.
Level 5 (or 6): Evaluation
Activities relating to using concepts to make judgments.
Continuing with the example of the arithmetic mean, shallow understanding could be demonstrated by reliance on heuristics and decision rules.
For example, when dealing with employee salaries, one might assume that the median is necessarily a superior representation of a “typical value,” because salaries tend to be positively skewed.
But, as with all rules of this nature, it’s not so simple…
Deeper understanding would take into account the nature of the particular situation or problem.
For example, suppose the organization has narrow salary bands due to limitations by law or agreements with labor unions. If so, then the salaries might not be skewed much or at all! In that case, the median becomes no better than the mean (or possibly worse, depending on what you’re doing with it).
It should not always be assumed that a variable must be distributed in a characteristic way (even if it usually is).
Additionally, even if there truly IS positive skew, that doesn’t necessarily eliminate the relevance of the mean.
For example, suppose you are analyzing compensation for a commission-based sales team. Imagine that most sales representatives on the team have comparable pay, but a few all-stars substantially increase their commissions year-after-year.
- If you rely solely on the median to represent average team performance, this trend could be completely concealed.
- But relying solely on the mean could lead to a different incorrect interpretation—that everyone on the team is improving, when it’s really just a few.
Here’s a quick demonstration of this using R…
> # Creating a toy dataset> typical=c(80,84,86,88,90,92)> stars=c(150,170)> (df=data.frame(id=101:108,+ year1=c(typical,stars),+ year2=c(typical,stars*2),+ year3=c(typical,stars*4))) id year1 year2 year31 101 80 80 802 102 84 84 843 103 86 86 864 104 88 88 885 105 90 90 906 106 92 92 927 107 150 300 6008 108 170 340 680> # The totals are increasing year after year...> colSums(df[2:4])year1 year2 year3 840 1160 1800 > # But the median does not reflect this!> (medians=apply(df[2:4],2,median))year1 year2 year3 89 89 89 > # The mean DOES reflect this...> # ...but it's biased by the all-stars!> (means=apply(df[2:4],2,mean))year1 year2 year3 105 145 225
The above demonstration shows that the median gives a more “typical” experience, whereas the mean reflects the overall group pay more accurately. That said, the latter is easily misunderstood; it paints a rosier picture than reality, because the change is only driven by a couple high performers.
With a distribution this small, the solution might be simply to view all scores simultaneously, in a table or graph.
For any number of scores, you could also combine the two types of averages to get a fuller picture…
> # Ratio of mean/median year over year...> data.frame(medians,means,ratio=round(means/medians,2)) medians means ratioyear1 89 105 1.18year2 89 145 1.63year3 89 225 2.53
The above table makes it clear that median pay is constant, while mean pay is rising (which suggests some subset of team members are getting increasingly higher pay).
Again, what all this illustrates is that simplistic rules and heuristics like “Don’t use the mean when the distribution is skewed” are the domain of shallow understanding.
Deeper understanding allows for considering what numbers are truly most relevant for a particular situation.
Level 6 (or 5): Creation (Synthesis)
Activities relating to combining concepts to create superordinate ideas or broader systems.
Shallow understanding could be demonstrated by creating an HR analytics dashboard that includes various summary statistics for relevant variables, including means.
Now, that’s not necessarily shallow. But it doesn’t inherently convey depth, either.
With basic knowledge of key terms and formulas, as well as familiarity with coding or a platform like Tableau, this can be accomplished even by someone who has no idea whether the mean is relevant for the variables and situation at hand.
One example of a task requiring deeper understanding is designing a job applicant scoring system that involves averaging ratings across a panel of interviewers, where it is known that one interviewer gives systematically harsher ratings than the others.
At first blush, you might think that the more sophisticated approach would involve some kind of correction, like a trimmed mean. (In fact, I know a guy with a PhD who once suggested this solution to this exact problem.)
This solution seems deeper, in that it doesn’t rely on canned statistical functions, and it takes into account the influence of skew…
But in a typical selection system, the overarching goal is to rank-order applicants and make job offers to those highest-ranked.
Critically, if a single rater is systematically more severe than other raters, this does NOT change the rank-order of applicants. (Assuming the same panel rates all interviewees.)
This is because systematically lenient or severe ratings are like adding or subtracting a constant (on average) to each score, which “moves them” all in the same way, and therefore has no effect on rank.
Consequently, you don’t need to make any adjustment at all! You can preserve all the information that the panel members provided about the applicants—and therefore avoid inflating average scores—while still getting the same applicant rankings.
Here’s a quick demonstration in R …
> # Creating a dataset of job applicant ratings> (ratings=data.frame(+ Candidate=c('Applicant A','Applicant B','Applicant C','Applicant D'),+ Rater1=c(4,3,5,2),+ Rater2=c(5,3,4,2),+ Rater3=c(4,4,5,1),+ Rater4_harsh=c(2,1,3,0))) Candidate Rater1 Rater2 Rater3 Rater4_harsh1 Applicant A 4 5 4 22 Applicant B 3 3 4 13 Applicant C 5 4 5 34 Applicant D 2 2 1 0> # Rater 4 is systematically harsher than others> # Rank-ordering applicants> rank(rowMeans(ratings[,2:5]))[1] 3 2 4 1> # Rankings without the harsh rater> rank(rowMeans(ratings[,2:4]))[1] 3 2 4 1> # The rankings are unchanged...
This shows that corrections like trimming extreme scores (in this case, literally eliminating the harsh rater’s scores from the dataset) have no effect on the actual applicant rankings.
And I can tell you from experience that this is far from obvious to most people, including many who have taken multiple statistics courses.
Now, this is not to say that corrections would never be appropriate. The point is just that choosing the right solution requires deep conceptual understanding of how averages like the arithmetic mean work.
Real-world applications require greater depth of understanding than mere definitions, formulas, and canned code.
All right, that’s all six levels. Hopefully I’ve made it painfully clear that depth of understanding does NOT necessarily rise along with successive levels of Bloom’s Taxonomy.
A reasonable question at this point might be, “Okay, so exactly how deep should we expect a given student’s understanding to extend?”
And I guess the answer must be individualized to the learner, instructor, training curriculum, university, and/or other context.
More generally, How deep does anyone’s understanding need to extend?
As I’ve mentioned previously, it’s relatively easy to identify shallow understanding, as well as to distinguish shallow from deep…
But it’s never clear how deep deep can be. How far down the rabbit hole can you go? Does it ever bottom out?
I don’t know. For now, I’ll keep digging.
