KDD 2015 Keynote
Online Controlled Experiments:
Lessons from Running A/B/n Tests for 12 years
Ron Kohavi
Abstract
The Internet provides developers of connected software, including web sites, applications, and devices, an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using trustworthy controlled experiments (e.g., A/B tests and their generalizations). From front-end user-interface changes to backend recommendation systems and relevance algorithms, from search engines (e.g., Google, Microsoft’s Bing, Yahoo) to retailers (e.g., Amazon, eBay, Netflix, Etsy) to social networking services (e.g., Facebook, LinkedIn, Twitter) to Travel services (e.g., Expedia, Airbnb, Booking.com) to many startups, online controlled experiments are now utilized to make data-driven decisions at a wide range of companies. While the theory of a controlled experiment is simple, and dates back to Sir Ronald A. Fisher’s experiments at the Rothamsted Agricultural Experimental Station in England in the 1920s, the deployment and mining of online controlled experiments at scale (e.g., hundreds of experiments run every day at Bing) and deployment of online controlled experiments across dozens of web sites and applications has taught us many lessons. We provide an introduction, share real examples, key lessons, and cultural challenges.
To reference keynote talk:
Ron Kohavi. 2015. Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '15). ACM, New York, NY, USA, 1-1. http://bit.ly/KDD2015Kohavi. DOI=http://dx.doi.org/10.1145/2783258.2785464
Photos, thanks to Carmen Lee from encapture:
Geoff Webb and Thorsten Joachims, the KDD 2015 program co-chairs
People voting on A/B Quiz
Few people remained standing: worse than random voting
Dianne Tang
DOI=http://dx.doi.org/10.1145/2783258.2785464
@inproceedings{Kohavi:2015:OCE:2783258.2785464,
author = {Kohavi, Ron},
title = {Online Controlled Experiments: Lessons from Running A/B/N Tests for 12 Years},
booktitle = {Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining},
series = {KDD '15},
year = {2015},
isbn = {978-1-4503-3664-2},
location = {Sydney, NSW, Australia},
pages = {1--1},
url = {http://bit.ly/KDD2015Kohavi},
doi = {10.1145/2783258.2785464},
acmid = {2785464},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {a/b testing, controlled experiments},
}