Oct. 8, 2026

Part of why LLMs are often (currently) terrible at evaluating whether statistical assumptions are met is that the underlying training data come from the far reaches of the Internet, where there’s a LOT of bad information about checking assumptions.

Many people online advise checking assumptions using significance tests (p-values). I’m actually not opposed to p-values in general, but I don’t think they’re very useful in the context of checking assumptions…

The main problem is that statistical significance is a function of sample size, so the tests are more likely to be significant in larger samples and less likely to be significant in smaller samples.

This is not great, since the assumptions are generally less critical with larger sample sizes and more critical with smaller sample sizes. In other words, significance tests tend to be most useful precisely when we don’t need them! (I know that’s a headache, but it’s worth thinking through.)

As always, genuine understanding of how these procedures actually work is essential to making effective decisions. Bad decisions result when people use tools they don’t understand.

Okay, so what’s the alternative?

Among valid approaches to checking assumptions, one of the most popular is to use graphs that show how well the data align with the assumptions.

So it’s a form of eyeball analysis. (Not a technical term.)

I know that doesn’t sound very scientific, but it absolutely is. It will get you closer to the truth than relying on a flowchart will.

Granted, this is more subjective than significance tests, but it’s also less arbitrary. The main limitation is this is only possible if you possess a deep understanding of the assumptions and the model more generally.

One thing I recommend doing to build your intuition for what a reasonable and expected deviation from the assumptions looks like (as opposed to a blatant violation) is to simulate data.

Run many simulations of what the data might look like when the assumptions do hold. Plot the results, and then compare your actual observations to the simulated ones.

If your actual data look like they could have been drawn from the simulated data, then that’s evidence that the assumptions do hold (at least approximately).

If your data sticks out like a sore thumb among the simulated data, then that’s evidence that the assumptions aren’t met.

The more data you see (real and simulated), the clearer your sense of all this will become. You can only get expert-level judgment by putting in lots and lots of reps.

As always, either (1) build the expertise yourself or (2) get someone on your team who already has it.