A/B Testing Intuition Busters:
Common Misunderstandings in Online Controlled Experiments

By Ron Kohavi, Alex Deng, and Lukas Vermeer

Appears in KDD 2022.  PDF

Abstract
A/B tests, or online controlled experiments, are heavily used in industry to evaluate implementations of ideas. While the statistics behind controlled experiments are well documented and some basic pitfalls known, we have observed some seemingly intuitive concepts being touted, including by A/B tool vendors and agencies, which are misleading, often badly so. Our goal is to describe these misunderstandings, the “intuition” behind them, and to explain and bust that intuition with solid statistical reasoning. We provide recommendations that experimentation platform designers can implement to make it harder for experimenters to make these intuitive mistakes.

Relevant posts

• LinkedIn announcement
• Tweet

 

ACM Reference format:
Ron Kohavi, Alex Deng, Lukas Vermeer. A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments.
In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22), August 14-18, 2022,
Washington DC, USA. https://doi.org/10.1145/3534678.3539160

Quick link: https://bit.ly/expIntuitionBusters