KDD 2015 Keynote

Online Controlled Experiments:
Lessons from Running A/B/n Tests for 12 years

Ron Kohavi

Talk slides in PPT and in PDF

DOI: 10.1145/2783258.2785464

Talk Video

 

Abstract

The Internet provides developers of connected software, including web sites, applications, and devices, an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using trustworthy controlled experiments (e.g., A/B tests and their generalizations). From front-end user-interface changes to backend recommendation systems and relevance algorithms, from search engines (e.g., Google, Microsoft’s Bing, Yahoo) to retailers (e.g., Amazon, eBay, Netflix, Etsy) to social networking services (e.g., Facebook, LinkedIn, Twitter) to Travel services (e.g., Expedia, Airbnb, Booking.com) to many startups, online controlled experiments are now utilized to make data-driven decisions at a wide range of companies. While the theory of a controlled experiment is simple, and dates back to Sir Ronald A. Fisher’s experiments at the Rothamsted Agricultural Experimental Station in England in the 1920s, the deployment and mining of online controlled experiments at scale (e.g., hundreds of experiments run every day at Bing) and deployment of online controlled experiments across dozens of web sites and applications has taught us many lessons. We provide an introduction, share real examples, key lessons, and cultural challenges.

 

 

To reference keynote talk:

 

Ron Kohavi. 2015. Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '15). ACM, New York, NY, USA, 1-1. http://bit.ly/KDD2015Kohavi.  DOI=http://dx.doi.org/10.1145/2783258.2785464

 

Photos, thanks to Carmen Lee from encapture:

Geoff Webb and Thorsten Joachims, the KDD 2015 program co-chairs

People voting on A/B Quiz

Few people remained standing: worse than random voting

Dianne Tang

DOI=http://dx.doi.org/10.1145/2783258.2785464

@inproceedings{Kohavi:2015:OCE:2783258.2785464, 

author = {Kohavi, Ron}, 

title = {Online Controlled Experiments: Lessons from Running A/B/N Tests for 12 Years}, 

booktitle = {Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining}, 

series = {KDD '15}, 

year = {2015}, 

isbn = {978-1-4503-3664-2}, 

location = {Sydney, NSW, Australia}, 

pages = {1--1}, 

url = {http://bit.ly/KDD2015Kohavi}, 

doi = {10.1145/2783258.2785464}, 

acmid = {2785464}, 

publisher = {ACM}, 

address = {New York, NY, USA}, 

keywords = {a/b testing, controlled experiments}, 

}