Osborn Learning & Analytics, LLC

Tag: modeling

  • Better Ways to Check Assumptions

    Oct. 8, 2026

    Part of why LLMs are often (currently) terrible at evaluating whether statistical assumptions are met is that the underlying training data come from the far reaches of the Internet, where there’s a LOT of bad information about checking assumptions.

    Many people online advise checking assumptions using significance tests (p-values). I’m actually not opposed to p-values in general, but I don’t think they’re very useful in the context of checking assumptions…

    The main problem is that statistical significance is a function of sample size, so the tests are more likely to be significant in larger samples and less likely to be significant in smaller samples.

    This is not great, since the assumptions are generally less critical with larger sample sizes and more critical with smaller sample sizes. In other words, significance tests tend to be most useful precisely when we don’t need them! (I know that’s a headache, but it’s worth thinking through.)

    As always, genuine understanding of how these procedures actually work is essential to making effective decisions. Bad decisions result when people use tools they don’t understand.

    Okay, so what’s the alternative?

    Among valid approaches to checking assumptions, one of the most popular is to use graphs that show how well the data align with the assumptions.

    So it’s a form of eyeball analysis. (Not a technical term.)

    I know that doesn’t sound very scientific, but it absolutely is. It will get you closer to the truth than relying on a flowchart will.

    Granted, this is more subjective than significance tests, but it’s also less arbitrary. The main limitation is this is only possible if you possess a deep understanding of the assumptions and the model more generally.

    One thing I recommend doing to build your intuition for what a reasonable and expected deviation from the assumptions looks like (as opposed to a blatant violation) is to simulate data.

    Run many simulations of what the data might look like when the assumptions do hold. Plot the results, and then compare your actual observations to the simulated ones.

    If your actual data look like they could have been drawn from the simulated data, then that’s evidence that the assumptions do hold (at least approximately).

    If your data sticks out like a sore thumb among the simulated data, then that’s evidence that the assumptions aren’t met.

    The more data you see (real and simulated), the clearer your sense of all this will become. You can only get expert-level judgment by putting in lots and lots of reps.

    As always, either (1) build the expertise yourself or (2) get someone on your team who already has it.

  • What if the Assumptions Aren’t Met?

    Oct. 7, 2026

    If you check the statistical assumptions of your analysis and find that they’re violated in your particular situation, that means you’re fitting the wrong model to the data.

    A slight or mild violation of the assumptions might not be anything to worry about. It might not change the results discernibly, or it might mean the results are only approximately correct.

    It’s ultimately a question of what level of precision you need for your business situation.

    • If you don’t care whether the conclusions from the data are even true, then you don’t need to check statistical assumptions at all!
    • If a close approximation of the truth is sufficient for your business situation, then you might choose to permit mild violations.
    • If you need to make high-precision predictions or base critical business decisions on the conclusions, then the assumptions are as important as the findings themselves.

    Yep. This gets pretty subjective and nuanced.

    The truth is that the assumptions are never perfectly met, because the assumptions describe purely theoretical models.

    (It’s the same concept as how there’s no perfect circle in physical reality. But, like the abstract concept of circles, our statistical models are incredibly useful, when used appropriately.)

    So it’s aways a question of how much of a violation of the assumptions you’re willing to permit.

    This requires careful and experienced judgment. And unfortunately it’s an area where AI is (currently) pretty terrible at making judgments.

    There are two ways to ensure these judgments are made correctly:

    1. Get a lot of experience looking at both real-life data and simulated data.
    2. Get someone stats-oriented on your team.

  • Check if You Should Check Assumptions

    Oct. 6, 2026

    Here’s an idea: If you’re skeptical about the importance of checking statistical assumptions, then try running some sensitivity tests.

    Run your analyses with and without assumption checks..

    1. First, run the analysis using whatever heuristic you normally use.
    2. Then fit a model rigorously, ensuring that its assumptions hold.
    3. Compare the results.

    Do this many times.

    Is there a meaningful difference in the utility for your business situation?

    • Do you invariably come to the same conclusions either way?
    • Do you always calculate nearly identical predictions either way?
    • Do you find the same levels of precision either way?

    If your conclusions flip, predictions diverge substantially, and/or precision fluctuates, then you have empirically demonstrated why we bother with checking assumptions.

  • “In the Real World…” We Don’t Check Assumptions?

    Oct. 5, 2026

    It’s no secret that in many applied data analytics contexts, formal steps like checking model assumptions are sometimes skipped.

    If this is admitted with a blush indicating the appropriate level of embarrassment, that’s one thing.

    It’s not great, but at least they’re aware that they have no idea if their findings are correct (or even interpretable).

    What’s more concerning is when the statement is delivered confidently, as though to suggest that such stuffy procedures are not really necessary “in the real world.”

    If that’s your stance, I have terrible news… You have no idea if your findings are correct (or even interpretable).

    What are statistical assumptions?

    They are NOT things we assume to be true.

    They’re more like conditionals, as in, “If these conditions are true, then these results and conclusions will be correct.”

    This means you can’t escape or ignore them. They’re part of the description of the model that you fit to the data.

    And there are no escape hatches like large datasets or AI. I’m not aware of any statistical procedure that makes zero assumptions—not even analyses for big data or machine learning algorithms.