06.7 · Capability in Measurement

Evidence-based decisions

Turning results into decisions: what the evidence really says, and what to do next.

Map4 skills

Not startedLearningComfortable

4 skillsClick a hexagon to set your status

06.7.1Interpreting resultsReading experiment and analysis results without fooling yourself.
06.7.2Causal reasoningTelling correlation from causation in product data.
06.7.3Balancing signalsCombining numbers, user feedback and judgment into one decision.
06.7.4Iterate, expand or stopDeciding what to do with a feature after you have results.

Mark all 4

Where to start

3 resources

articleGo deeper
A/B testing intuition busters: common misunderstandings in online controlled experiments ↗
Ron Kohavi, Alex Deng, Lukas Vermeer · 2022 · 10 pages
Why surprising wins are usually wrong, why low-powered tests exaggerate effects, and how to read p-values correctly.
articleGo deeper
How and when to control for confounders during product usage analyses ↗
Heap · 2021 · 10 min
Uses the familiar "users of this feature retain better" case to show how a confounder fakes an effect and how to adjust for it.
bookGo deeper
Causal inference for the brave and true ↗
Matheus Facure · free online book
A free, rigorous and readable book on causal inference with Python code, from randomized tests to methods for when you cannot run one.

0 tools

Tools for this topic are being researched.