Online Controlled Experiments and A/B Tests

By Ron Kohavi and Roger Longbotham

Appears in Encyclopedia of Machine Learning and Data Science 2023, edited by Dinh Phung, Geoff Webb, and Claude Sammut.

Online Controlled Experiments and A/B Tests (PDF)

Definitive version:



ACMRef: Ron Kohavi and Roger Longbotham, 2023. Online Controlled Experiments and A/B Tests. In: Phung, D., Webb, G.I., Sammut, C. (eds) Encyclopedia of Machine Learning and Data Science. Springer, New York, NY.


© 2023. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive version is published in the Encyclopedia of Machine Learning and Data Mining.



Many good resources are available with motivation and explanations about online controlled experiments (Kohavi et al. 2009a, 2020; Thomke 2020; Luca and Bazerman 2020; Georgiev 2018, 2019; Kohavi and Thomke 2017; Siroker and Koomen 2013; Goward 2012; Schrage 2014; King et al. 2017; McFarland 2012; Manzi 2012; Tang et al. 2010). For organizations running online controlled experiments at scale, Gupta et al. (2019) provide an advanced set of challenges. We provide a motivating visual example of a controlled experiment that ran at Microsoft’s Bing. The team wanted to add a feature allowing advertisers to provide links to the target site. The rationale is that this will improve ads quality by giving users more information about what the advertiser’s site provides and allow users to directly navigate to the sub-category matching their intent. Visuals of the existing ads layout (Control) and the new ads layout (Treatment) with site links added are shown in Fig. 1.