A/B tests
Comparing two versions with real users to see which performs better.
An A/B test shows each version to a random group and compares a metric. It is the most reliable way to measure the effect of a product change, given enough traffic. With low traffic, other methods often teach more.
Resources
2- articleStart hereHow not to run an A/B test ↗Evan Miller · 2010 · 8 min
The classic explanation of why peeking at results and stopping early produces false wins, and what to do instead.
- bookGo deeper · PaidTrustworthy online controlled experiments: a practical guide to A/B testing ↗Ron Kohavi, Diane Tang, Ya Xu · 2020 · 288 pages
The standard reference on running experiments at scale, written by the people who built experimentation at Microsoft, Google and LinkedIn.
Tools
3- ToolOpen sourceGrowthBook
Open-source feature flags and A/B testing that analyzes results from your own data warehouse.
- ToolFreemiumStatsig
Feature flags, experiments and product analytics with built-in statistics such as variance reduction.
- ToolOpen sourcePostHog
Open-source product analytics with session replay, feature flags, experiments and surveys in one tool.