Osborn Learning & Analytics, LLC

Tag: knowledge

  • Do We Understand p-Values? (Pt. 1)

    Nov. 2, 2026

    Originally, this post was going to be about how to use calculus to calculate p-values. This would be yet another demonstration of how we can use digital representations of our mental models (i.e., code) to test how well we understand a concept.

    However, that explanation relies on the reader already understanding p-values at a relatively deep level.

    So, instead, this post became an introduction to p-values…

    (This is the verbal expression of my conceptualization; next time, we’ll convert this into a digital representation.)

    Few statistical concepts are as misunderstood (or maligned) as the mythical p-value.

    Many people who frequently use p-values don’t really understand them. For example, a substantial number have simply memorized that “If p < .05, it’s statistically significant,” which they use as a decision rule.

    But while those who hate p-values usually do understand them fairly well, the uniform disapproval of their use reflects a different kind of confusion.

    There is nothing wrong with p-values—so long as you don’t misuse them.

    What’s a p-value?

    In the context of statistical significance testing, a p-value is simply the proportion (p) of a null distribution that is at least as extreme as an observed result. (Nothing inherently good or bad about it.)

    Okay, but what does THAT mean?

    Essentially, a significance test is about signal detection. Most of the time, the question is, “Is there any effect at all?” Or, put another way, “Does it seem like the effect is different than literally zero?”

    (It’s debatable how informative that question is, as well as whether there aren’t better questions we could be asking, but let’s keep walking through it for now.)

    In this context, an effect refers to an association between two variables. For example, is there any link between supervisor quality and intent to quit your job? We might check by gathering some data and calculating a statistic like a correlation.

    • If there’s no link whatsoever, then the effect is zero.
    • If there is some link, whether positive or negative, whether large or small, then the effect is NOT zero.

    A significance test evaluates this question using a backwards sort of logic:

    1. Assume (hypothetically) that the effect is literally zero.
    2. Calculate the result we’d expect to see if this were true.
    3. Repeat this calculation many, many times. (Due to randomness, the results will vary somewhat.)
    4. Pile up all the calculated results into a distribution (i.e., a null distribution).
    5. Check where your observed result lies within this distribution.

    (Fortunately, we don’t actually have to run an infinite number of analyses. We can use math and/or simulations to derive this null distribution.)

    In this way, you are comparing the result of your actual analysis to the various possible results that would be expected due to mere randomness.

    This is a signal-to-noise evaluation…

    • If your result is similar to what would be expected due to random noise, you might interpret that as evidence* favoring zero effect.
    • If your result is very different from what would be expected due to random noise, you might interpret that as evidence* favoring a nonzero effect.

    *NOTE: I broadly define the term evidence to mean any piece of information that sways one’s credence toward a particular belief. In this way, I’m essentially using a p-value as a (very) loose proxy for a Bayes Factor, if you know what that is. (Not getting into that right now.)

    Regardless, note that this is NOT how some statisticians use the term “evidence.” And that’s fine. Words are just labels for concepts, so if we explain what we mean by them, we can use them however we like.

    But how do you determine how compatible your result is with random noise?

    This determination is based on where your observed result lies relative to the rest of the distribution…

    The shape of the null distribution describes how relatively likely different results are (assuming the true effect is zero). In locations where the distribution piles up higher, those results are more likely.

    Therefore, the nearer your observed result lies to the densest mass of the distribution, the more compatible it is with zero effect. That’s because your finding is the sort of result that would be predicted under zero effect.

    In contrast, the further your observed result lies from the mass of the distribution, the less compatible it is with zero effect…because your finding is NOT the sort of result that would be predicted under zero effect.

    So, how to quantify the position of the observed result relative to the rest of the distribution?

    This is where the p-value comes in: it is simply the proportion of results in the null distribution that are at least as extreme as the actual result of your analysis.

    For example, if your result is way far away from the main mass of the null distribution, that proportion will be quite small (say, p = .00003). In contrast, if your result is near the main mass, the proportion will be relatively larger (say, p = .47).

    So, all things being equal, a larger p-value is more compatible with the notion of zero effect (i.e., the data resemble what random noise would produce)…

    And a smaller p-value is less compatible with the notion of zero effect (i.e., the data do NOT resemble what random noise would produce).

    It is extremely commonplace to treat this as a binary evaluation:

    • If p < .05, the result is “statistically significant.”
      • (A result at least this extreme would occur less than 5% of the time if the true effect is zero.)
    • If p ≥ .05, the result is “NOT statistically significant.”
      • (A result at least this extreme would occur at least 5% of the time if the true effect is zero.)

    In my opinion, this binary interpretation is a bad idea. It lends itself to overly simplistic decision rules, which are nearly always indicative of shallow understanding. It also eliminates the majority of the information that the p-value itself provides.

    Instead, I interpret p-values as what they literally are: The probability of a result at least as extreme as our analysis showed, if we assume the true effect is zero.

    I like to imagine a p-value as a slider… As p gets smaller, your credence (level of belief) in zero effect should fall; slide it back the other way, and your credence should rise again.

    But note that this slider only indicates direction, not precise amount. The extent to which your credence should adjust based on a p-value depends on various other factors, such as sample size, data quality, your previous beliefs, etc.

    In this way, a p-value can be interpreted as some amount of evidence for or against the notion of no effect. But it’s up to the individual to judge exactly how persuasive that evidence is.

    The binary decision rule seems to serve primarily as a mechanism to avoid individual judgment. That’s a cognitive trap.

    Understood correctly, there’s nothing inherently good or bad about p-values. They are simply a property of a result, assuming certain conditions hold.

    A person’s interpretation of what that value means can certainly be wrong. But the value is what it is.

    Next time, we’ll check if this explanation holds up by converting it into digital form (i.e., code), and then testing it.

  • Using Calculus to Find the Area of a Triangle

    Nov. 1, 2026

    The area of a triangle can be found with (1/2) base ⋅ height, which makes sense because a right triangle is half of a rectangle with area base ⋅ height.

    Hopefully, that formula already seems intuitively clear.

    But we can also find the area of a triangle using calculus…

    I like to think of calculus as geometry for weird shapes, but it can also be used to find the area of commonplace shapes like squares, triangles, and circles.

    One way to think of integration is as a method of slicing up a shape into many thin strips, where the width of each strip approaches 0. In other words, you’re slicing the shape as thinly as possible.

    The area of each slice can then be approximated by a rectangle with a width approaching zero and a height determined from the dimensions of the original shape.

    The area of the shape can therefore be found by summing the areas of all these thin slices.

    For example, consider a triangle with base = 5 and height = 10. We know from the basic geometric formula that the area is (1/2) 5 ⋅ 10 = 25.

    If we “sliced” this into vertical strips, the leftmost strip would have a height of 0, and the rightmost strip would have a height of 10. If we summed the areas of all the vertical slices from height 0 to height 10 (i.e., from x = 0 to x = 5), the total would be 25.

    We can check whether our understanding of integration is correct using R code.

    There are multiple ways to do this, but I’m going to approximate the correct answer by literally slicing the triangle into many very thin strips.

    Specifically, I’m going to cut it into 50 million slices. (That’s a lot, but R can do it in a split-second.)

    > # Triangle Dimensions
    > base=5
    > height=10
    > # Standard Area Check
    > (area=base*height/2)
    [1] 25
    > # Specifying 50 million slices
    > slices=seq(0,5,1e-07)

    Now, how do we approximate the area of each slice? Each width has been defined as 1e-07 = .0000001.

    If we examine the triangle, we’ll notice that the height of each vertical strip is twice its position along the x-axis (i.e., the distance from the left end of the triangle). For example…

    1. When x = 0, the height is 0
    2. When x = 2.5, the height is 5
    3. When x = 5, the height is 10

    Another way of saying this is that the top edge of the triangle is a line described by y = 2x (the slope is 2).

    So, now let’s identify the height of each slice….

    > # Heights of slices
    > slice_heights=slices*2

    And then combine the heights and widths into areas of each slice…

    > # Areas = Heights * Widths
    > slice_areas=slice_heights * 1e-07

    And the last step is to sum the areas of these slices…

    > sum(slice_areas)
    [1] 25

    Note that we only get precisely 25 because R displays just 7 significant digits by default. If we instruct R to be more precise, we’ll see that the calculation is not quite 25…

    > # This is less precise than it seems...
    > print(sum(slice_areas),digits=15)
    [1] 25.0000005

    But the narrower the slices we use, the more the result will dial in on precisely 25.

    Okay. I think the concept has been demonstrated well enough.

    Clearly, there are much more rigorous ways to approach this kind of thing. And I’m sure that any actual mathematician reading this would fly into a rage at the imprecision in my explanations and demonstrations.

    But the point is we’ve written code to represent our understanding of the concept (integration), and this code accurately produced expected results to an acceptable degree of approximation.

    Next time, I’ll apply the same concept to show how we calculate these magical p-values that stats folks are always talking about…

  • Using Code to Represent Deeper Understanding

    Oct. 31, 2026

    I believe everyone should learn to code, because it allows for creating digital representations of the concepts that exist in our minds, and this allows for testing them for accuracy.

    In the previous post, however, I showed that a coded representation of an idea can still be incredibly shallow. In that example, I showed that creating custom R code for the arithmetic mean could simply reflect a basic formulaic understanding: “Add up the parts, divide by the number of parts.”

    A deeper level of understanding might involve realizing that the arithmetic mean is only one specific case of a set of general functions (e.g., geometric mean, harmonic mean, exponential mean)…

    For all these functions, there is some operation (e.g., addition, multiplication, etc.) conducted on each value in the distribution, and then they are aggregated in a way that shows what an “equal contribution” of each would be.

    For example, with the arithmetic mean, we are adding all the parts up. And the mean represents the single value that could replace every datapoint in the distribution while still summing to the same total as the original dataset. In this way, it’s a sort of “typical” score in an additive sense.

    Suppose we had 3 scores: (1, 2, 3)…

    • The arithmetic mean is (1 + 2 + 3)/3 = 2.
    • The sum of the original dataset is 1 + 2 + 3 = 6
    • If we replaced each value with the mean (2), we would get precisely the same sum: 2 + 2 + 2 = 6

    So this family of means (which is infinitely large) says: If you perform some operation on all the scores, then the mean will be the value that could replace each without changing the result of the operation. In other words, it shows what an “equal contribution” to the operation would be.

    If this understanding is correct, then we should be able to write a function in R (or whatever programming language you prefer) that takes as its inputs (1) a set of scores and (2) the operation being performed, and then produces the resulting mean.

    Let’s take 4 examples from this family of means…

    1. Arithmetic Mean = Σx/n (Operation: Addition)
    2. Geometric Mean = ⁿ√(Πx) (Operation: Multiplication)
    3. Harmonic Mean = n / Σ(1/x) (Operation: Addition of reciprocals)
    4. Quadratic Mean = √(Σx²/n) (Operation: Addition of squares)

    Pulling this off successfully will require understanding a more general algorithm that binds all these (and all others in this family) together…

    Geometric Mean

    When each value in a set is multiplied, the arithmetic mean produces misleading results.

    Example: Suppose you invest $100. If your return in the first year is +30% (i.e., 130% of original) and then your return in the second year is -30% (i.e., 70% of Year 1), then what is the average growth?

    The arithmetic mean would seem to suggest 0% growth…

    • Σx/n = (1.30 + (.70)) / 2 = 1
      • (In this case, 1 means 100% of the original, so zero return overall)

    But this is obviously incorrect if we check step-by-step…

    1. Year 1: $100 ⋅ 1.30 = $130
    2. Year 2: $130 ⋅ .70 = $91

    We’ve lost $9…so clearly it’s not 0% growth.

    The better solution is the geometric mean, which gives the typical multiplicative value…

    • ⁿ√(Πx) = ²√(1.30 ⋅ .70) ≈ .9539

    We could replace each year’s return with this value and then get the correct ending value…

    1. Year 1: $100 ⋅ .9539 = $95.39
    2. Year 2: $95.39 ⋅ .9539 ≈ $91

    So what’s the link between the arithmetic mean and the geometric mean?

    Well, the real question is, “What’s the link between addition and multiplication?” And the answer is logarithms.

    Specifically, log(a·b) = log(a) + log(b).

    This means that if we first take the log of each value, we can proceed to use the ordinary arithmetic mean…

    • [ln(1.3) + ln(.7)] / 2 ≈ -0.0472

    …then exponentiate (i.e., return to original units) to get the correct answer!

    • e^-0.0472 ≈ .9539

    This is the same result we got when using the geometric mean formula.

    Harmonic Mean

    When working with reciprocals, the ordinary arithmetic mean again fails…

    Example: If you drive at 50 mph on the way to your destination, then you drive back at 40 mph, then what was your average speed?

    It may seem intuitively clear that the answer is 45 mph (the midpoint between 40 and 50). That value is the arithmetic mean… and it is WRONG.

    That calculation would only work if you traveled for the same time units each time (e.g., 50 mph for 1 hour and then 40 mph for 1 hour).

    But you were traveling 10 mph faster on the way to the destination, so it did NOT take you the same amount of time each way.

    The real solution is the harmonic mean, which expresses the calculation in terms of the thing that is constant (i.e., distance to and from the destination). It’s as simple as flipping the miles / hour fraction…

    • (1/50 + 1/40) / 2 = .0225 hours per mile (hpm?)

    …then taking the reciprocal to put it back in the original units…

    • 1 / .0225 ≈ 44.44 mph

    Quadratic Mean

    When working with square units (e.g., tiles on a bathroom floor), the naive arithmetic mean again fails…

    Example: Suppose you’ve got 100 tiles to cover the floor of a room. Half are 6 inches across, and half are 8 across. What’s the average coverage of a given tile? And how much area will this cover?

    It might be tempting to just think, “Well the average tile is 7 inches across (i.e., the arithmetic mean of 6 and 8), so that’s 7² = 49 sq in. Multiply times 100 tiles to get 4,900 sq in.”

    Wrong, wrong, wrong…

    The 6″ tiles have area 6² = 36 sq in, and the 8″ tiles have area 8² = 64 sq in. The arithmetic mean is (36 + 64) / 2 = 50 sq in. And that multiplied times 100 tiles is 5,000 sq in.

    The quadratic mean, √(Σx²/n), handles this correctly. First, calculate the mean of the squares…

    • Σx²/n = [50⋅6² + 50⋅8²] / 100 = 50 sq in

    …And then we can take the square root to convert back to original units (inches rather than square inches)…

    • √50 ≈ 7.07107 in

    The average tile’s side length is 7″… but that does not accurately reflect the total tile area!

    (By the way, I’ve never laid tile before, so I don’t know if being off by this factor would be a serious problem or not. But I personally would want the correct calculation.)

    Quasi-Arithmetic Mean

    In case you haven’t noticed it yet, the general process is…

    1. Transform the scores
    2. Calculate the arithmetic mean: Σx/n
    3. Transform the scores back into original units (inverse transformation)

    The general form of this process is called the quasi-arithmetic mean…

    • Mean = f ⁻¹ [Σf(x) / n]

    (This formula is nothing new. It’s just expressing that 3-step process above using mathematical notation.)

    The transformation (f) could be taking the logarithm (for the geometric mean), taking the reciprocal (for the harmonic mean), taking the square (for the quadratic mean), and so on.

    Transform, calculate the arithmetic mean, then transform back to original units.

    Finally, we’re ready to write R code that takes any set of scores as input, along with a specification regarding the type of operation conducted, and that then spits out the resulting mean (whether arithmetic, geometric, harmonic, or whatever).

    Here’s a version where each step is explicitly laid out…

    > # Custom Mean Function
    > generalized_mean=function(data,transformation,inverse) {
    +   t_data=transformation(data) # Transformation
    +   m_t_data=sum(t_data)/length(t_data) # Arithmetic mean calculation
    +   result=inverse(m_t_data) # Inverse transformation
    +   return(result)
    + }
    > # Example 1: Arithmetic Mean
    > x1=1:3
    > generalized_mean(x1,identity,identity)
    [1] 2
    > # Example 2: Geometric Mean
    > x2=c(1.3,.7)
    > generalized_mean(x2,log,exp)
    [1] 0.9539392
    > # Example 3: Harmonic Mean
    > x3=c(50,40)
    > generalized_mean(x3,function(x) 1/x, function(x) 1/x)
    [1] 44.44444
    > # Example 4: Quadratic Mean
    > x4=c(rep(6,50),rep(8,50))
    > generalized_mean(x4,function(x) x^2, sqrt)
    [1] 7.071068

    All the results match our above calculations.

    But we can make it WAY cleaner by collapsing the function into a single line representing f ⁻¹ [Σf(x) / n]…

    > # Cleaner Generalized Mean Code
    > gen_mean_clean=function(x,f,inv_f) {inv_f(mean(f(x)))}
    > # Arithmetic
    > gen_mean_clean(x1,identity,identity)
    [1] 2
    > # Geometric
    > gen_mean_clean(x2,log,exp)
    [1] 0.9539392
    > # Harmonic
    > gen_mean_clean(x3,function(x) 1/x, function(x) 1/x)
    [1] 44.44444
    > # Quadratic
    > gen_mean_clean(x4,function(x) x^2, sqrt)
    [1] 7.071068

    If we check our work using the standard formulas, we get the exact same answers…

    > # Checking our work...
    > mean(x1)
    [1] 2
    > prod(x2)^(1/length(x2))
    [1] 0.9539392
    > length(x3)/sum(1/x3)
    [1] 44.44444
    > sqrt(sum(x4^2)/length(x4))
    [1] 7.071068

    In each of the 4 cases, the mean finds the value that would constitute an “equal contribution” to some result, namely…

    1. The sum of scores for the arithmetic mean
    2. The product of scores (i.e., sum of log-scores) for the geometric mean
    3. The sum of reciprocals (e.g., total time) for the harmonic mean
    4. The sum of squares (e.g., total area) for the quadratic mean

    If we can follow all of that, then I think it’s safe to say we have fairly deep understanding of the concept of the mean.

  • Code Can Reflect Shallow Understanding

    Oct. 30, 2026

    Previously (here and here), I’ve argued that everyone should learn to code (even truck drivers in 2026).

    The core concept is that code allows for creating digital representations of our internal mental models, which allows us to test and confirm whether our understanding is actually correct.

    But the fact that you can represent your understanding with code does not necessarily mean that you possess deep understanding.

    It’s entirely possible to write code that reflects shallow understanding.

    I’ll explore an example below, using a basic Stat 101 concept, the arithmetic mean (what is often meant by the term average).

    A shallow understanding of the mean might be demonstrated by describing it as, “The result of adding up the parts and dividing by the number of parts.”

    The definition is true, but it’s very basic, and one could utter these exact words in this sequence without really knowing what it means.

    We can represent this level of understanding using R code in a fairly straightforward way…

    > # Sample Data
    > (x=1:6)
    [1] 1 2 3 4 5 6
    > # Mean Formula from Scratch
    > seth_mean=function(data) {sum(data)/length(data)}
    > # Let's see if it works...
    > seth_mean(x)
    [1] 3.5
    > # It matches the result of the Base R mean function
    > mean(x)
    [1] 3.5

    We can try to break the new code, such as by messing with the inputs. For example, my custom function has a failure mode relating to NA values…

    > # Sample Data with a missing value
    > (y=c(x,NA))
    [1]  1  2  3  4  5  6 NA
    > seth_mean(y)
    [1] NA
    > # The Base R function can handle this...
    > # ...if we simply add an argument
    > mean(y,na.rm=T)
    [1] 3.5

    Note that the above approach “handles” the missing data by omitting those cases, which is not necessarily the best approach.

    Reagrdless, I can modify my custom function to accommodate missing values (say, by omitting them), maybe like this…

    > # Modified function that omits NA values
    > seth_mean2=function(data) {
    +   data=na.omit(data)
    +   sum(data)/length(data)
    + } 
    > # Testing it out...
    > seth_mean2(y)
    [1] 3.5

    But what extent of understanding does this code actually convey?

    It could represent something deep… or it could merely reflect understanding equivalent to, “Uhh…it’s a statistic that you get when you add things up and divide by the number of them.”

    Next time, I’ll explore using code to represent a deeper level of understanding…

  • Code as Falsifiable Externalization of Understanding

    Oct. 29, 2026

    I dropped yesterday’s post (Everyone Should Learn to Code) into Claude just to see what it would say, and it commented, among other things…

    “Your idea is: code as a falsifiable externalization of understanding.”

    And as much as I usually hate AI writing, even I have to admit that’s a pretty clean summarization of what I’m trying to describe.

    I phrased it more like this: Coding allows us to create digital representations of concepts in our minds, which we can then test to see if they hold up.

    In one sense, this is more robust than the mental representations themselves, which aren’t directly subject to external verification.

    In this post, I’ll provide a simple demonstration of this concept: Deriving the formula for the diagonal of a cube.

    Along the way, I’m going to outline some ground rules for how to do it effectively…

    1. Your code has to be built up “from scratch” and incrementally.
    2. Your code has to come directly from your own mental model.
    3. Your code has to be verified using an independent method.
    4. Your code has to survive your best efforts to “break” it.

    1. Build the code from scratch.

    This is similar to first principles thinking, which basically entails breaking ideas down into their most fundamental parts (each of which seem axiomatically true to you) and then re-combining the parts.

    A key benefit of this approach is that it allows you to avoid memorizing a bunch of complicated information; instead, grasping a few basic principles enables you to combine them into more complex forms in the moment.

    Here’s a simple illustration: Offhand, I don’t recall the formula for the diagonal of a unit cube.

    (I remember it involves a square root or a cube root or something, but I don’t bother memorizing things like this. As this exercise shows, there’s no need to.)

    However, I do know the Pythagorean Theorem (PT), which can be used to derive the diagonal of a cube.

    Critically, I know the PT at an intuitive, visual level. I haven’t just memorized it; I know it virtually must be true (p > .99999), based on having seen dozens of proofs. I can’t imagine how it would be possible for it not to be true. So, for this exercise, we’ll accept the PT an axiom or first principle.

    Okay, let’s see if we can work out what the diagonal of a unit cube would be… (note that this is a verbal representation)

    • By definition, each side of a unit cube is a square, and each edge has length 1 (let’s call each edge e, so e = 1).
    • When I visualize a cube in my mind (or draw one), I see that the biggest diagonal (not sure what it’s called, let’s call it D) cuts across the middle of the cube from one corner to the opposite one.
    • In my visualization, I notice D forms the hypotenuse of a right triangle, where one side is an edge of the cube (e), and the other is the diagonal of one of the square-sides (let’s call this smaller diagonal d).
    • I already know e = 1, but I need to find d. In my visualization, I can see that d is the hypotenuse of another right triangle, formed by d and two sides of that square. Therefore, we can use the PT to find this smaller diagonal…
      • e² + e² = d²
      • (1)² + (1)² = d²
      • 2 = d²
      • d = √2
    • Now, I can again apply the PT to find the hypotenuse of the larger triangle…
      • e² + d² = D²
      • (1)² + (√2)² = D²
      • 1 + 2 = D²
      • 3 = D²
      • D = √3

    (Okay, now I know why I was thinking there might be a “cube root”…it’s the square root of 3.)

    Now, I’m sure mathematicians could show much more efficient ways to derive the formula, but that would risk missing the point.

    The point here is that I can build up that higher-level concept in the moment, based on more basic concepts like the PT. My goal is understanding, not speed.

    All right. The task now is to write code that represents this concept.

    There are various ways I could do this, but I’m going to write an R function that finds the diagonal of any cube I define.

    > # This function takes the edge of any cube...
    > # ...and outputs the "big diagonal"
    > cube_diag=function(edge=1) {
    +   d=sqrt(2*edge^2)
    +   D=sqrt(d^2+edge^2)
    +   return(D)
    + }
     
    > # By default, it assumes a unit cube (edge = 1)
    > cube_diag()
    [1] 1.732051
     
    > # Note that this is equal to √3...
    > sqrt(3)
    [1] 1.732051
     
    > # We can also scale the cube up, say to edge = 2...
    > cube_diag(2)
    [1] 3.464102
    > # So D scales proportionally with the edges...
    > cube_diag(2) / cube_diag()
    [1] 2
    > cube_diag(3) / cube_diag()
    [1] 3

    Clearly, I could simply treat D = e⋅√3 as a memorized fact, which is faster and more efficient…

    > cube_diag_fast=function(edge=1) {edge*sqrt(3)}
    > # This gives the same results with less code...
    > cube_diag_fast(1:2)
    [1] 1.732051 3.464102

    But, again, that would be missing the point.

    If we just wanted the answer, we could have Googled it from the start. The purpose is to create and verify a digital representation of our understanding.

    Some Caveats…

    Note 1: I don’t mean that ALL code should be built up from scratch. That would be incredibly inefficient for getting work done on a daily basis.

    I do think you should build anything you use from scratch the first time or two that you use it. But once you understand something intuitively, it becomes far more efficient to automate.

    Understand, then automate.

    Note 2: A reasonable question is, “Exactly how far do we need to break things down?” For example, should a person be able to represent these concepts with pure 0s and 1s? (I hope not, because I can’t do that.)

    The answer is subjective, but in my view, anything that you already grasp intuitively and visually can be kept “whole” in the code you write.

    For example, if you already have an intuitive grasp of the concept of addition, then it’s fine to just use the sum() function (or whatever the equivalent function is in your programming language).

    Note 3: I like to dispense with fancy, technical terms when possible. Those terms are important and useful, but the more I use them, the higher the risk that I accidentally end up relying on a concept by name only, without necessarily grasping the underlying idea.

    So, yeah…that’s my justification for calling it the “big diagonal.”

    (I looked it up just now, and it’s actually called a space diagonal or similar.)

    2. Build the code from your own mental model.

    If you copy code from another source, or if you regurgitate an explanation you heard someone else give, this does NOT demonstrate independence, which is a core feature of how I define deep understanding (explained here).

    To be fair, I’m sure I originally learned this solution, or at least something similar, from someone else a long time ago (geometry class?). But if so, then I’d forgotten it.

    I solved it today by visualizing the cube and its diagonal in my mind, independently of any formula or diagram I’ve seen. (Of course, this did rely on my remembering terms like cube and diagonal.)

    And I wrote the code myself, in a way that made sense to me. That said, I should definitely check it against an LLM or another source to make sure I didn’t screw anything up…

    3. Verify the code using an independent method.

    When independent processes come to the same conclusion for any sufficiently complex task, the probability of the result being incorrect decreases.

    I looked online, and it turns out there is another version of the formula that I had forgotten about: D = √(e² + e² + e²).

    (This jogs my memory about how we can extend into as many dimensions as we want [like hypercubes], and the diagonal will always be calculated by adding another e² under the square root for every additional dimension. But that’s getting off-track.)

    Let’s use this “separate” (though algebraically equivalent) method to confirm our calculations…

    > # Alternative (equivalent) Formula
    > cube_diag2=function(edge=1) {sqrt(edge^2+edge^2+edge^2)}
    > # It gives the same results as our other formula...
    > cube_diag2(1:2)
    [1] 1.732051 3.464102

    Claude also suggested identifying coordinates for a cube in geometric space and using the dist() function (for the distance formula).

    > # Using the Distance Formula
    > max(dist(expand.grid(x=c(0,1),y=c(0,1),z=c(0,1))))
    [1] 1.732051
    > max(dist(expand.grid(x=c(0,2),y=c(0,2),z=c(0,2))))
    [1] 3.464102

    It all matches up.

    I guess we could also use measurements of an actual physical box to confirm as well, at least to an approximation. But I think I’ll pass on that…

    4. Try to break the code.

    The goal here is to ensure that your representation is falsifiable. You want to seek out disconfirming evidence (prove yourself wrong), and if you fail, that makes you more confident (though not certain) that you understand correctly.

    To do this, let’s input some silly “edge lengths”…

    > # Let's see what kinds of results we get for differing inputs...
    > cube_diag(0)
    [1] 0
    > cube_diag(-1)
    [1] 1.732051
    > cube_diag(1e200)
    [1] Inf

    Clearly, our code representation has some assumptions about what lengths we’re dealing with.

    1. The code works just fine with lengths of 0, even though there is no cube (real or imagined) that contains no space.
    2. The code also works for negative edge lengths, which doesn’t make sense for a cube.
    3. The code concludes that a very large cube has an infinitely long diagonal, which is incorrect.

    So, if we wanted to employ this function more broadly, we’d need to add more specifications to avoid potential incorrect output.

    But overall, I’d say the code accomplishes its purpose.

    This exercise makes me more confident that I do understand the concept of a diagonal of a cube, including how it relates to the other properties of the shape.

    And this mental representation in my mind exists independently of formulas found online, textbook definitions, or other people’s explanations.

    Up next:

    1. Demonstrating shallow understanding of the mean
    2. Demonstrating deeper understanding of the mean

  • Deep Bloom (Pt. 3)

    Oct. 26, 2026

    In Part 1, I compared shallow and deep understanding Levels 1 and 2 of Bloom’s Taxonomy (Knowledge and Comprehension, respectively, to use the original terminology).

    In Part 2, I did the same for Levels 3 and 4 (Application and Analysis).

    In this final installment, I do the same for Levels 5 and 6 (Evaluation and Synthesis).

    Note that the names of Bloom’s levels, as well as the order, are different depending on which version you’re looking at, but it doesn’t really matter.

    Level 5 (or 6): Evaluation

    Activities relating to using concepts to make judgments.

    Continuing with the example of the arithmetic mean, shallow understanding could be demonstrated by reliance on heuristics and decision rules.

    For example, when dealing with employee salaries, one might assume that the median is necessarily a superior representation of a “typical value,” because salaries tend to be positively skewed.

    But, as with all rules of this nature, it’s not so simple…

    Deeper understanding would take into account the nature of the particular situation or problem.

    For example, suppose the organization has narrow salary bands due to limitations by law or agreements with labor unions. If so, then the salaries might not be skewed much or at all! In that case, the median becomes no better than the mean (or possibly worse, depending on what you’re doing with it).

    It should not always be assumed that a variable must be distributed in a characteristic way (even if it usually is).

    Additionally, even if there truly IS positive skew, that doesn’t necessarily eliminate the relevance of the mean.

    For example, suppose you are analyzing compensation for a commission-based sales team. Imagine that most sales representatives on the team have comparable pay, but a few all-stars substantially increase their commissions year-after-year.

    • If you rely solely on the median to represent average team performance, this trend could be completely concealed.
    • But relying solely on the mean could lead to a different incorrect interpretation—that everyone on the team is improving, when it’s really just a few.

    Here’s a quick demonstration of this using R…

    > # Creating a toy dataset
    > typical=c(80,84,86,88,90,92)
    > stars=c(150,170)
    > (df=data.frame(id=101:108,
    +                year1=c(typical,stars),
    +                year2=c(typical,stars*2),
    +                year3=c(typical,stars*4)))
       id year1 year2 year3
    1 101    80    80    80
    2 102    84    84    84
    3 103    86    86    86
    4 104    88    88    88
    5 105    90    90    90
    6 106    92    92    92
    7 107   150   300   600
    8 108   170   340   680
    > # The totals are increasing year after year...
    > colSums(df[2:4])
    year1 year2 year3 
      840  1160  1800 
     
    > # But the median does not reflect this!
    > (medians=apply(df[2:4],2,median))
    year1 year2 year3 
       89    89    89 
     
    > # The mean DOES reflect this...
    > # ...but it's biased by the all-stars!
    > (means=apply(df[2:4],2,mean))
    year1 year2 year3 
      105   145   225

    The above demonstration shows that the median gives a more “typical” experience, whereas the mean reflects the overall group pay more accurately. That said, the latter is easily misunderstood; it paints a rosier picture than reality, because the change is only driven by a couple high performers.

    With a distribution this small, the solution might be simply to view all scores simultaneously, in a table or graph.

    For any number of scores, you could also combine the two types of averages to get a fuller picture…

    > # Ratio of mean/median year over year...
    > data.frame(medians,means,ratio=round(means/medians,2))
          medians means ratio
    year1      89   105  1.18
    year2      89   145  1.63
    year3      89   225  2.53

    The above table makes it clear that median pay is constant, while mean pay is rising (which suggests some subset of team members are getting increasingly higher pay).

    Again, what all this illustrates is that simplistic rules and heuristics like “Don’t use the mean when the distribution is skewed” are the domain of shallow understanding.

    Deeper understanding allows for considering what numbers are truly most relevant for a particular situation.

    Level 6 (or 5): Creation (Synthesis)

    Activities relating to combining concepts to create superordinate ideas or broader systems.

    Shallow understanding could be demonstrated by creating an HR analytics dashboard that includes various summary statistics for relevant variables, including means.

    Now, that’s not necessarily shallow. But it doesn’t inherently convey depth, either.

    With basic knowledge of key terms and formulas, as well as familiarity with coding or a platform like Tableau, this can be accomplished even by someone who has no idea whether the mean is relevant for the variables and situation at hand.

    One example of a task requiring deeper understanding is designing a job applicant scoring system that involves averaging ratings across a panel of interviewers, where it is known that one interviewer gives systematically harsher ratings than the others.

    At first blush, you might think that the more sophisticated approach would involve some kind of correction, like a trimmed mean. (In fact, I know a guy with a PhD who once suggested this solution to this exact problem.)

    This solution seems deeper, in that it doesn’t rely on canned statistical functions, and it takes into account the influence of skew…

    But in a typical selection system, the overarching goal is to rank-order applicants and make job offers to those highest-ranked.

    Critically, if a single rater is systematically more severe than other raters, this does NOT change the rank-order of applicants. (Assuming the same panel rates all interviewees.)

    This is because systematically lenient or severe ratings are like adding or subtracting a constant (on average) to each score, which “moves them” all in the same way, and therefore has no effect on rank.

    Consequently, you don’t need to make any adjustment at all! You can preserve all the information that the panel members provided about the applicants—and therefore avoid inflating average scores—while still getting the same applicant rankings.

    Here’s a quick demonstration in R …

    > # Creating a dataset of job applicant ratings
    > (ratings=data.frame(
    +   Candidate=c('Applicant A','Applicant B','Applicant C','Applicant D'),
    +   Rater1=c(4,3,5,2),
    +   Rater2=c(5,3,4,2),
    +   Rater3=c(4,4,5,1),
    +   Rater4_harsh=c(2,1,3,0)))
        Candidate Rater1 Rater2 Rater3 Rater4_harsh
    1 Applicant A      4      5      4            2
    2 Applicant B      3      3      4            1
    3 Applicant C      5      4      5            3
    4 Applicant D      2      2      1            0
    > # Rater 4 is systematically harsher than others
    > # Rank-ordering applicants
    > rank(rowMeans(ratings[,2:5]))
    [1] 3 2 4 1
    > # Rankings without the harsh rater
    > rank(rowMeans(ratings[,2:4]))
    [1] 3 2 4 1
    > # The rankings are unchanged...

    This shows that corrections like trimming extreme scores (in this case, literally eliminating the harsh rater’s scores from the dataset) have no effect on the actual applicant rankings.

    And I can tell you from experience that this is far from obvious to most people, including many who have taken multiple statistics courses.

    Now, this is not to say that corrections would never be appropriate. The point is just that choosing the right solution requires deep conceptual understanding of how averages like the arithmetic mean work.

    Real-world applications require greater depth of understanding than mere definitions, formulas, and canned code.

    All right, that’s all six levels. Hopefully I’ve made it painfully clear that depth of understanding does NOT necessarily rise along with successive levels of Bloom’s Taxonomy.

    A reasonable question at this point might be, “Okay, so exactly how deep should we expect a given student’s understanding to extend?”

    And I guess the answer must be individualized to the learner, instructor, training curriculum, university, and/or other context.

    More generally, How deep does anyone’s understanding need to extend?

    As I’ve mentioned previously, it’s relatively easy to identify shallow understanding, as well as to distinguish shallow from deep…

    But it’s never clear how deep deep can be. How far down the rabbit hole can you go? Does it ever bottom out?

    I don’t know. For now, I’ll keep digging.

  • Deep Bloom (Pt. 2)

    Oct. 25, 2026

    Part 1 compared shallow and deep understanding for the first two levels of Bloom’s Taxonomy. This post covers the next two levels, again using the mean (average) as the example.

    Level 3: Application

    Activities relating to using concepts to solve novel problems.

    Shallow understanding at this level would involve simply plugging numbers into an algorithm to produce the mean.

    For example, I might use the following R code to find the mean for the values (1, 2, 3, 4, 5, 6).

    > mean(1:6)
    [1] 3.5

    Clearly, this shows one can apply some superficial concept of the mean (e.g., “it’s an important number I can get with the mean() function”) to solve novel problems without necessarily having any idea what it actually means.

    A contrived problem like this could be made trickier to test deeper understanding.

    For example, suppose that a class of 24 students has an average exam score of 85, and a separate class of 31 students has an average score of 82. What’s the average score across the two classes?

    Without grasping the underlying concept, the first thought might be to average the averages: (85 + 82)/2 = 83.5.

    But this is WRONG.

    As mentioned last time, the arithmetic mean can be conceptualized as an equal share per unit. In other words, if we totaled all the scores in the distribution, then cut the total into equal parts, the size of each equal part would be the mean.

    But the combined classes do NOT have 2 scores, so that’s the wrong denominator; the combined count is 24 + 31 = 55. We need to account for that in our calculation.

    (There’s a quick formula to solve for a weighted mean, but going straight to that from memory would again be missing the point here. That approach could also be employed formulaically, without really understanding what’s going on at a conceptual level.)

    If we had all the individual scores, we could just ignore the individual class means and calculate the overall. But since we only have summary data, we’ll have to reverse-engineer it…

    The mean for one class is found with M₁ = Σx₁/n₁. That means that we could multiply the mean times the count to get total: Σx₁ = M₁n₁. We can do this for both groups… Σx₁ = 85 ⋅ 24 = 2040 and Σx₂ = 82 ⋅ 31 = 2542.

    Now we can find the total sum and the total count for the two groups combined: Combined Mean = (Σx₁ + Σx₂) / (n₁ + n₂) = (2040 + 2542) / (24 + 31) ≈ 83.31.

    In fact, the process we just walked through can be formalized as Weighted Mean = (M₁n₁ + M₂n₂) / (n₁ + n₂). This is the quick formula I mentioned previously, but, again, simply memorizing it would bypass the underlying intuition.

    If someone genuinely understands the concept of the mean at a deep level, then they could figure this out on their own, without needing to be taught the formula separately.

    Level 4: Analysis

    Activities relating to breaking concepts down into sub-components.

    Shallow understanding, at the most basic level, would involve simply recognizing and grasping the two components of the mean formula:

    1. The sum of the values, Σx
    2. The number of values, n

    It might follow from this that outlying values can drag the mean around, which might constitute deeper learning. But there is also a version of such “understanding” that is merely a memorized rule: The mean is dragged in the direction of outliers.

    If someone told me that, I’d be tempted to ask, “Why? What makes it work that way? Does this happen in some lawful way?”

    And those are harder questions to answer…

    Deeper understanding could come from examining the formula more closely…

    The formula Σx/n could be rewritten as x₁/n + x₂/n + … + xₙ/n. This provides an alternative interpretation of the mean as a weighted sum of scores, where each score gets a weight of 1/n (meaning each score counts equally).

    As such, each score contributes xᵢ/n to the mean. That is, every score contributes in proportion to its own value (but the influence is scaled by the total number of scores).

    This is why every score (including outlying values) “pulls the mean” in its own direction. The further a given score, xᵢ, lies from the other values, the more it changes the mean from what it would be without xᵢ.

    To illustrate this, let’s see what would happen if we changed the value of a single score…

    Increasing a given score by Δ increases the total sum of the distribution by exactly Δ. Therefore, taking the weight (1/n) into account, the mean must increase by Δ/n.

    For example, consider this set of scores: (0, 1, 2, 3, 4). If we increase that first value from 0 to 500 (i.e., Δ = 500), it will increase the total sum by 500, and it will increase the mean by Δ/n = 500/5 = 100.

    Here’s a quick demonstration in R…

    > # Create a distribution with 5 scores...
    > x=c(0,1,2,3,4)
    > sum(x)
    [1] 10
    > mean(x)
    [1] 2
    > # Increase the first value by 500
    > x[1]=x[1]+500
    > # Now the sum has increased by 500...
    > sum(x)
    [1] 510
    > # And the mean has increased by 500/5 = 100...
    > mean(x)
    [1] 102

    (We would also see the opposite pull downward if Δ were negative.)

    But, by the same token, this also demonstrates how the extent of any single value’s influence is controlled by the number of scores.

    In the above example (n = 5), the Δ for that first score changed the mean dramatically. But if we have 50,000 scores, the same increase has almost no effect…

    > # Create a distribution with 50k scores...
    > set.seed(123)
    > x=sample(1:5,50000,T)
    > sum(x)
    [1] 150075
    > mean(x)
    [1] 3.0015
    > # Increase the first value by 500
    > x[1]=x[1]+500
    > # Now the sum has increased by 500...
    > sum(x)
    [1] 150575
    > # But the mean has only increased by 500/50000 = .01...
    > mean(x)
    [1] 3.0115

    In this way, all values in the distribution “drag the mean around,” but this influence is most apparent when (1) a given score is further away from the others and (2) there are relatively few datapoints in the distribution.

    And this is just one example of how breaking the concept down into component parts could lead to a deeper understanding—in this case through rearranging the formula to get a different intuition.

    Okay, once again, this ended up being longer than expected. Final installment tomorrow…

  • Deep Bloom (Pt. 1)

    Oct. 24, 2026

    In a previous post, I gave examples of how a person could understand the mean (average) across all six levels of Bloom’s Taxonomy.

    But I later argued that Bloom only gives us categories of activities relating to knowledge, not necessarily levels of increasing complexity or depth.

    To more clearly illustrate what I mean, below are examples of both shallow and deeper understanding at each level of Bloom…

    Level 1: Remembering (Knowledge)

    Activities relating to identifying and recalling concepts.

    Sticking with the example of the mean (average)…

    Shallow understanding can be demonstrated by writing out a basic version of the formula, Σx/n, or saying something like, “Add up the parts, then divide by the number of parts.”

    While this is accurate, a person could write or utter these words without necessarily being able to clarify what any of it actually means.

    Deeper understanding in this level might constitute memorizing a network of concepts within which the mean sits.

    For example, you might know that this is only one type of average among many, including…

    • Mean
    • Median
    • Mode

    In fact, it’s not even the only mean out there. This category includes…

    • Arithmetic mean (what people usually mean by “average”)
    • Geometric mean
    • Harmonic mean

    At this level of Bloom, deeper understanding could include the ability to identify and recall definitions and formulas for each of these and others.

    It could also include knowing different or more complex variations of the formula, which type of average is most appropriate for different types of analytic situations, and so on…

    Ultimately, however, all these terms, as well as their definitions and formulas, are just words, which are merely labels for concepts.

    As such, while it would be difficult to imagine someone having deep understanding of these concepts without knowing the relevant terms, that knowledge alone is still not sufficient to guarantee comprehension.

    Imagine you memorized a brief passage in a foreign language by writing and speaking it daily…

    At some point, you’d have it down pat. But unless you ever bothered to translate it, you’d never know what any of it meant!

    In the same way, a person could memorize every relevant term and definition in a given subject domain until they could recite it all at will, and that still wouldn’t guarantee that any of it meant anything to them.

    Level 2: Understanding (Comprehension)

    Activities relating to explaining concepts.

    Shallow understanding here might be demonstrated by an explanation like, “The mean is a typical or average score in a distribution.”

    This is true, but it’s incredibly vague. It seems to overlook the existence of other types of averages (see previous section). There’s also no apparent realization that certain factors (e.g., outliers, distribution shape…) could affect how well it typifies the distribution.

    Two common candidate explanations for deeper understanding would be…

    1. The mean as the balancing point of the distribution; half the “weight” lies below it, and half lies above it. This is why the sum of deviations from the mean is always zero.
    2. The mean as the equal share per unit; if we totaled the scores, then cut the total into equal parts (one for each observation in the distribution), the value of each part would be the mean.

    This is usually how I introduce the concept to students.

    But we could go even deeper.

    Perhaps the deepest (?) level of any mathematical or statistical concept is the algorithmic level.

    Here’s the idea: If you truly understand a concept, you should be able to create your own algorithm from scratch that reproduces the idea. (Doesn’t really matter whether it’s unique or not, so long as you generate it yourself.)

    One way to take the arithmetic mean to this level of understanding is to realize it’s only one case of a more general process. There are more technical versions of this, but here’s the basic generalization…

    Suppose you perform some operation (e.g., addition, multiplication) on a set of values, resulting in a total effect. For example, summing all the values results in a total; multiplying all the values results in a product; et cetera.

    A mean is a single value that could replace each score in the distribution without changing the result of that operation.

    Consider summation (adding things up)…

    If our set of scores is [1, 2, 3, 4], the result of summation would be 1 + 2 + 3 + 4 = 10. The arithmetic mean, Σx/n = 2.5, is the only value that could replace each score in that operation, while still producing the same result: 2.5 + 2. 5 + 2. 5 + 2.5 = 10.

    Or consider multiplication…

    The product of multiplying all the values in that same distribution is 1 ⋅ 2 ⋅ 3 ⋅ 4 = 24. Again, the geometric mean, ⁿ√(Πx) ≈ 2.2134, is the only value that could replace each score in that operation, while producing the same result: 2.2134 ⋅ 2.2134 ⋅ 2.2134 ⋅ 2.2134 ≈ 24.

    What makes this explanation “deeper”?

    To me, it’s deeper in that it represents a more general principle about how things work. The mean is not (just) an arbitrary concept that people use by convention; it fits into the bigger picture of how math works.

    Based on this, it becomes apparent that there are infinite number of possible means that we could define.

    This allows for understanding an entire class of calculations with a single concept. For example, someone who understands the above explanation is in a far better position to next grasp the harmonic mean and various other means.

    Viewed through this lens, the arithmetic mean is only one case of a single value that can represent a distribution of scores in a particular context. It’s not some magical statistic that we should always default to when summarizing a set of scores. (This has critical implications for other Bloom levels, like application, creation, etc.).

    Another takeaway from this discussion is that deeper understanding can often be acquired by exploring different ways of explaining the same concept.

    The more multidimensional your understanding of a concept, the more your mental representation of it begins to transcend any particular set of symbols on a page.

    And it doesn’t have to stop here.

    Yet another alternative explanation could be through a regression lens:

    Using an ordinary least squares approach, the arithmetic mean is the predicted value of the distribution when no additional variables are taken into consideration (i.e., no predictors; intercept-only model).

    By “predicted value,” I mean one possible “best guess” for what a randomly selected score from the distribution would be.

    And on and on it goes…

    Okay, this post is already longer than I expected it to be. More next time…

  • Shallow Mastery

    Oct. 22, 2026

    Last time, I argued that Bloom’s Taxonomy does NOT necessarily represent progressively advanced levels of learning or understanding.

    For one thing, a task in the Remembering/Knowledge level could be extremely sophisticated (e.g., recall the names of all bones in the human body, as well as the origins of the names). Similarly, a task in the Evaluating level could be relatively basic (e.g., judge whether an x-ray depicts a broken bone).

    For another, regardless of the complexity of the task at hand, the person’s extent of understanding can fall anywhere along a continuum from shallow to deep…

    Like the ocean, shallow areas of understanding can be easily spotted from the surface; a few penetrating questions can usually reveal that someone doesn’t really know what they’re talking about.

    In contrast, deeper understanding is harder to gauge. Even when we can recognize that someone’s understanding isn’t shallow, the actual depth remains murky—as does the maximum possible depth in the topic, if there even is such a thing.

    Due to the way our brains work, we generally opt for shallow understanding. We tend to want the minimum viable understanding in order to be able to move on or apply the information.

    Consequently, a person could demonstrate “understanding” in all six levels of Bloom while still possessing very little depth of understanding in the relevant subject matter.

    To show this is true, consider anyone who has an advanced degree, yet understands very little of the subject with any depth. (I’ll give a generic example in a minute.)

    If you’re not familiar with academia, this may sound preposterous. But the truth is that many learners adopt a “learn-to-get-by” mentality, not a “learn-to-understand” mentality, and at many institutions that is enough to obtain an advanced degree.

    We can speculate about why this is the case, but that’s not the point here.

    Consider the fictional example of John Smith, who is a mashup of a variety of real people…

    John majored in psychology and statistics during college, then went on to get a masters degree in the social sciences with a quantitative specialization.

    (By the way, none of what follows is John’s fault, exactly—except insofar as he failed to claim responsibility for his own learning. The broader problem lies in how humans naturally think and how certain educational systems are designed.)

    What John was taught—and in some sense mastered—was a cookbook-style overview of statistics, where flowcharts guided all decision points and AI helped write code to conduct all analyses.

    For example, if a homework assignment asked John to compare the means of two independent groups, he looked at his flowchart and found that it instructed him to conduct an “independent samples t-test.” (In preparation for closed-book exams, John even memorized this If–Then rule in a basic pattern-matching way.)

    The AI confirmed the use of a t-test and provided R code to produce summary statistics [e.g., mean() ] and test results [e.g., t.test() ]. The results were interpreted according to complicated decision rules about what the numbers do or do not mean.

    Depending on the program of study, John might also have learned about statistical assumptions, which are in turn evaluated with their own bevy of procedures and decision rules.

    John followed all required steps, submitted the requested information, and was awarded an “A.” Many such instances, across a variety of different problem scenarios and respective procedures, resulted in successful graduation from the program.

    Notice that all six levels of Bloom’s Taxonomy have been addressed many times over. For example…

    1. Remembering/Knowledge: John accurately recalled the names of the appropriate statistical tests for a variety of data analysis situations, as well as the associated R functions to run the rests.

    2. Understanding/Comprehension: He successfully paraphrased various statistical concepts (e.g., a 95% confidence interval was described as, “The interval within which the population parameter would fall in 95% of repeated samples.”)

    3. Applying: He correctly ran countless analyses on homework assignments and exams according to flowcharts and decision rules.

    4. Analyzing: He parsed and interpreted statistical output (e.g., effect sizes were extracted and interpreted according to Cohen’s guidelines; p-values were extracted and interpreted as “statistically significant” or not according to an α = .05 threshold).

    5. Evaluating: He scrutinized published research papers during class sessions to determine (a) whether the authors ran the correct analyses; (b) if the reported results are correct; and (c) whether the conclusions align with the results.

    6. Creating/Synthesis:     He conducted his own research projects, analyzing the results and perhaps even publishing them in scientific journals!

    Despite all this, a few penetrating questions might cause it all to fall apart like a house of cards.

    • Why does the flowchart say to run an ANOVA rather than a t-test? Why is one test “right” and the other “wrong”? Why is that rule the rule?
    • Where does that p-value come from? Why does it tell us anything about how to think about the results? What’s the difference between p = .0499 and p = .0501?
    • If we were feeling particularly cruel, we might ask basic questions about degrees of freedom—a concept John would have learned about in the first week or two of the program, but which is rarely understood by either students or instructors!

    Even if a response to any of these questions could be mustered, a follow-up question of, “But what does that mean?” would probably be the death blow.

    Why would these basic questions stump John?

    Because the only depth that has been established is with respect to the flowcharts and decision rules, not the why and how of statistical mechanics. (And certainly not the underlying phenomena being studied.)

    So much of learning is skimming along the surface, weaving a web of tenuously connected concepts, where much of it is accepted without question.

    Again, this isn’t because John is stupid or unmotivated (necessarily). It’s because he’s faced with an infinite barrage of information from every direction, and he needs some way to prioritize and get through it all. He’s probably not even consciously aware he’s doing this.

    But this is what he’s doing if the answer to the question, “Why?” is, “Because that’s the rule.”

    Why do we care? If John can get the correct results, then does it really matter?

    It DOES matter, because this flimsy kind of understanding tends to fall apart in real-world analytic situations, which call not for flowcharts but judgment and nuance.

    Additionally, superficial understanding is more easily replaced by AI than deep understanding is, so there’s not much job security for John in the current age.

    Now, I know it may seem like I’m being really mean to John in this post…

    But the truth is that we are ALL John.

    We all have certain domains of knowledge in which we don’t really understand what’s going on with much depth, even if we’ve completed high school, college, trainings, etc.

    In many domains we don’t need deep understanding.

    I’m worse than John when it comes to car engines. I don’t even know where to start. But things have pretty much worked out so far, because that’s not my area.

    The question is, What’s the domain (or two or three) where you ARE going to have deep understanding?

  • Levels of Bloom’s Levels

    Oct. 21, 2026

    You might think that Bloom’s Taxonomy makes the concepts of learning or understanding complicated enough, but the model is actually more complicated than I previously let on—and I would argue that the reality is even more complicated than the full model suggests!

    Here’s the 2001 revision of the model, which still breaks learning down into 6 levels of generally increasing complexity…

    1. Remembering
    2. Understanding
    3. Applying
    4. Analyzing
    5. Evaluating
    6. Creating

    If you’re paying attention, you’ll notice that the 5th and 6th levels are swapped in the revised version. Additionally, the levels are now framed as verbs. (This is due to how educators write learning objectives, which is one of the primary uses of the taxonomy.)

    Previously, I described the levels in a general “overview” way. In actuality, scholars have broken down each of the six levels into sub-components. For example…

    • Remembering breaks down into identifying and recalling (think “matching” vs. “fill-in-the-blank” test questions).
    • Understanding breaks down into several dimensions, including classifying, comparing, explaining…
    • Evaluating breaks down into checking and critiquing.
    • Et cetera

    Additionally, the different levels can be mapped against different types of knowledge (e.g., factual vs. procedural).

    Here’s a 2D representation of this idea, with example tasks in each cell (courtesy of Claude):

    (For details on the actual revised model from the scholars themselves, check out Anderson & Krathwohl [2001].)

    But even within this more nuanced framework, I would argue that there are degrees of understanding at each level and its component dimensions, on a spectrum from shallow to deep.

    For example, in the Remembering/Knowledge level, consider a person’s recall of facts in a subject area. Their extent of recall could be…

    • Shallow (e.g., remember a few introductory key terms)
    • Deep (e.g., remember many advanced terms)
    • Or anywhere in between

    Or in the Understanding/Comprehension level, consider a person’s ability to explain some mechanical process, such as how a lightbulb works. Again, their extent of explanatory ability could be…

    • Shallow (e.g., vaguely describe the general process)
    • Deep (e.g., describe the process in great depth and in multiple ways)
    • Or anywhere in between

    This is called explanatory depth.

    I’m not the only person who has thought of this, by the way. For example, Biggs & Collis (1982) published a model of what is essentially depth of learning, which they map onto cognitive developmental stages.

    Additionally, the concept of explanatory depth is the subject of an entire research literature—the illusion of explanatory depth (i.e., the study of how we typically LACK the ability to explain causal processes with as much depth as we think we can).

    It doesn’t look as clean, but I had Claude add a third dimension to the previous image to represent depth of understanding. (Appropriately, if the other two axes represent length and height, then this third dimension represents literal depth.)

    This is an attempt to illustrate how different demonstrations of knowledge can occur with varying amounts of depth; therefore, it is not as simple as higher-order processes always representing greater depth of understanding.

    (Granted, I suspect that higher-order levels such as Creating generally do correspond to deeper understanding than lower-order levels such as Remembering; my argument is only that it’s not guaranteed.)

    In this way, I view Bloom’s Taxonomy merely as six types of activities relating to knowledge. This argument doesn’t make the framework any less useful, but it does reveal a clear problem…

    A person could technically demonstrate “understanding” in all six of Bloom’s levels while still having basically no idea what they’re talking about.

    Reflecting on my own learning process, as well as what I observe in others, it seems to me that “shallow mastery” is more common than we’d like to believe, and it’s far less common than deep understanding.

    More on this next time…