Osborn Learning & Analytics, LLC

Tag: statistics

  • Creating a Custom AI Statistics Tutor

    Oct. 11, 2026

    Asking AI to give you the answer is the worst idea you’ve ever had.

    But using AI as your personalized tutor, drawing specifically from high-quality source materials, is the future of learning.

    In this video, I demonstrate how to create a custom Gem (using Google Gemini) for statistics tutoring, using my own course materials as the reference material it draws from.

    Then I try (and repeatedly fail) to “jailbreak” it to make it give me an answer key or sample walkthrough for a homework assignment.

    Why do this? Why not just open any chatbot application, like ChatGPT.com or Claude.ai and work with that?

    That approach isn’t necessarily wrong, but it’s the most unsophisticated version of this. And if you’re a student in my class, the output is often going to get you a bad score on the assignment.

    It’s a common misconception that in statistics and data analytics there are “right answers.” This comes from the fact that it’s a field of applied math, and typically math has right answers.

    But the reality is that there’s an incredible amount of subjective judgment involved in real-world statistical analysis. That means AI can’t give you “THE” correct procedure.

    LLMs are essentially the distilled voice of the Internet, and that includes quite a bit of incorrect information and bad advice.

    We can somewhat overcome this limitation by using custom LLMs, which draw primarily from source materials of our own choosing.

    My custom AI Stats Tutor may not be as good an educator as I am, but it’s the next best thing (for my particular class).

    And unlike me, it doesn’t need to sleep, meaning my students can query it day or night and get instant responses!

    As with any technology, there’s incredible potential here, if it’s used correctly (and programmed thoughtfully).

  • Better Ways to Check Assumptions

    Oct. 8, 2026

    Part of why LLMs are often (currently) terrible at evaluating whether statistical assumptions are met is that the underlying training data come from the far reaches of the Internet, where there’s a LOT of bad information about checking assumptions.

    Many people online advise checking assumptions using significance tests (p-values). I’m actually not opposed to p-values in general, but I don’t think they’re very useful in the context of checking assumptions…

    The main problem is that statistical significance is a function of sample size, so the tests are more likely to be significant in larger samples and less likely to be significant in smaller samples.

    This is not great, since the assumptions are generally less critical with larger sample sizes and more critical with smaller sample sizes. In other words, significance tests tend to be most useful precisely when we don’t need them! (I know that’s a headache, but it’s worth thinking through.)

    As always, genuine understanding of how these procedures actually work is essential to making effective decisions. Bad decisions result when people use tools they don’t understand.

    Okay, so what’s the alternative?

    Among valid approaches to checking assumptions, one of the most popular is to use graphs that show how well the data align with the assumptions.

    So it’s a form of eyeball analysis. (Not a technical term.)

    I know that doesn’t sound very scientific, but it absolutely is. It will get you closer to the truth than relying on a flowchart will.

    Granted, this is more subjective than significance tests, but it’s also less arbitrary. The main limitation is this is only possible if you possess a deep understanding of the assumptions and the model more generally.

    One thing I recommend doing to build your intuition for what a reasonable and expected deviation from the assumptions looks like (as opposed to a blatant violation) is to simulate data.

    Run many simulations of what the data might look like when the assumptions do hold. Plot the results, and then compare your actual observations to the simulated ones.

    If your actual data look like they could have been drawn from the simulated data, then that’s evidence that the assumptions do hold (at least approximately).

    If your data sticks out like a sore thumb among the simulated data, then that’s evidence that the assumptions aren’t met.

    The more data you see (real and simulated), the clearer your sense of all this will become. You can only get expert-level judgment by putting in lots and lots of reps.

    As always, either (1) build the expertise yourself or (2) get someone on your team who already has it.

  • What if the Assumptions Aren’t Met?

    Oct. 7, 2026

    If you check the statistical assumptions of your analysis and find that they’re violated in your particular situation, that means you’re fitting the wrong model to the data.

    A slight or mild violation of the assumptions might not be anything to worry about. It might not change the results discernibly, or it might mean the results are only approximately correct.

    It’s ultimately a question of what level of precision you need for your business situation.

    • If you don’t care whether the conclusions from the data are even true, then you don’t need to check statistical assumptions at all!
    • If a close approximation of the truth is sufficient for your business situation, then you might choose to permit mild violations.
    • If you need to make high-precision predictions or base critical business decisions on the conclusions, then the assumptions are as important as the findings themselves.

    Yep. This gets pretty subjective and nuanced.

    The truth is that the assumptions are never perfectly met, because the assumptions describe purely theoretical models.

    (It’s the same concept as how there’s no perfect circle in physical reality. But, like the abstract concept of circles, our statistical models are incredibly useful, when used appropriately.)

    So it’s aways a question of how much of a violation of the assumptions you’re willing to permit.

    This requires careful and experienced judgment. And unfortunately it’s an area where AI is (currently) pretty terrible at making judgments.

    There are two ways to ensure these judgments are made correctly:

    1. Get a lot of experience looking at both real-life data and simulated data.
    2. Get someone stats-oriented on your team.

  • Check if You Should Check Assumptions

    Oct. 6, 2026

    Here’s an idea: If you’re skeptical about the importance of checking statistical assumptions, then try running some sensitivity tests.

    Run your analyses with and without assumption checks..

    1. First, run the analysis using whatever heuristic you normally use.
    2. Then fit a model rigorously, ensuring that its assumptions hold.
    3. Compare the results.

    Do this many times.

    Is there a meaningful difference in the utility for your business situation?

    • Do you invariably come to the same conclusions either way?
    • Do you always calculate nearly identical predictions either way?
    • Do you find the same levels of precision either way?

    If your conclusions flip, predictions diverge substantially, and/or precision fluctuates, then you have empirically demonstrated why we bother with checking assumptions.

  • “In the Real World…” We Don’t Check Assumptions?

    Oct. 5, 2026

    It’s no secret that in many applied data analytics contexts, formal steps like checking model assumptions are sometimes skipped.

    If this is admitted with a blush indicating the appropriate level of embarrassment, that’s one thing.

    It’s not great, but at least they’re aware that they have no idea if their findings are correct (or even interpretable).

    What’s more concerning is when the statement is delivered confidently, as though to suggest that such stuffy procedures are not really necessary “in the real world.”

    If that’s your stance, I have terrible news… You have no idea if your findings are correct (or even interpretable).

    What are statistical assumptions?

    They are NOT things we assume to be true.

    They’re more like conditionals, as in, “If these conditions are true, then these results and conclusions will be correct.”

    This means you can’t escape or ignore them. They’re part of the description of the model that you fit to the data.

    And there are no escape hatches like large datasets or AI. I’m not aware of any statistical procedure that makes zero assumptions—not even analyses for big data or machine learning algorithms.