06.5.3 · Skill in Statistical reasoning
Effect sizes
How big a difference is, not only whether it exists.
Statistical significance says an effect is probably real. Effect size says whether it matters. A tiny but significant lift can be worth less than the complexity it adds.
Resources
1- articleGo deeperA/B testing intuition busters: common misunderstandings in online controlled experiments ↗Ron Kohavi, Alex Deng, Lukas Vermeer · 2022 · 10 pages
Why surprising wins are usually wrong, why low-powered tests exaggerate effects, and how to read p-values correctly.
Tools
none yetTools for this topic are being researched.