Microsoft's Experimentation Platform

Accelerating software innovation through trustworthy experimentation
Home
Contact Us
Job Descriptions
Cool Things about Microsoft
ExP Articles
ExP Talks & Presentations
ExP Tools
What's a HiPPO?
KDD 2009 Tutorial: Planning, Running, and Analyzing Controlled Experiments on the Web
By Ronny Kohavi, Roger Longbotham, John Quarto-vonTivadar
 
At KDD 2009, we taught a tutorial.  The three parts are below.
  1. Controlled Experiments tutorial part 1: PPTX, PDF
  2. Controlled Experiments tutorial part 2: PPTX, PDF
  3. Controlled Experiments tutorial part 3: PPTX, PDF
 
 

Tutorial Description

The web provides an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called randomized experiments, A/B tests (and their generalizations), split tests, and MultiVariable Tests (MVT). Controlled experiments embody the best scientific design for establishing a causal relationship between changes and their influence on user-observable behavior. Data Mining and Knowledge Discovery techniques can then be used to analyze the data from such experiments. The tutorial will provide a survey and practical guide to running controlled experiments based on the recently published survey article in the Data Mining and Knowledge Discovery Journal, co-authored with the two of the tutorial co-presenters (http://exp-platform.com/dmkd_survey.aspx), and based on the book “Always Be Testing” co-authored by the 3rd tutorial co-presenter (http://www.amazon.com/Always-Be-Testing-Complete-Optimizer/dp/0470290633). The book includes use of industry tools, such as Google Website Optimizer and recently ranked #2 on Amazon’s sales rank for computers/e-commerce books. The tutorial includes multiple real-world examples of actual controlled experiments (many with surprising results), a review the theory and the statistics used to plan and analyze such experiments, and a discussion of the limitations and pitfalls that might face experimenters. 

 

A video of a related talk can be found on the videolectures website:
http://videolectures.net/cikm08_kohavi_pgtce/


The shorter version of the DMKD survey paper is now part of the class reading for several classes at Stanford University (CS147, CS376), USCD (CSE 291), and at the University of Washington (CSEP 510).

Topics covered include:

  1. Why online experimentation using controlled experiments is important
  2. What you need in order to conduct a valid experiment
  3. Planning and Analysis of basic experiments
  4. Benefits and limitations of experimentation
  5. Multivariable experiments: setup, analysis, interpretation, and interactions