Welcome to A/B Testing at Scale Tutorial

Tutorial by: Somit Gupta, Ron Kohavi, Alex Deng, Paul Raff
Given at Strata San Jose 3/2018.

Tutorial Outline

1. Introduction to A/B testing (slides)
   1. What is A/B testing
   2. Brief history
   3. Why use A/B testing
   4. Examples
   5. Cultural Challenges
2. Design of Experiments: Statistical Foundations (slides)
   1. Null-hypothesis testing, confidence intervals, p-values
   2. Central Limit Theorem
   3. Power analysis
3. Ensuring Trustworthiness and High Quality (slides)
   1. Importance and example issues
      1. Data Quality
      2. A/A Tests
      3. Sample Ratio Mismatch tests
      4. Dealing with carry-over effects and random imbalance
      5. Twyman’s law
4. Protecting Users (slides)
   1. Start small then ramp up
   2. Near-real-time detection and shut down of bad experiments
   3. Interaction prevention
   4. Interaction detection
5. Designing Metrics (slides)
   1. Data Quality Metrics
   2. OEC (Overall Evaluation Criteria)
   3. Guardrail Metrics
   4. Local Feature and Diagnostic Metrics
   5. Metric Interpretation Pitfalls
6. Active Research Areas and Recent Development (slides)
   1. Metric Sensitivity
   2. Issues with Null Hypothesis Testing and p-value
   3. Continuous Decision Making
   4. Beyond Average Treatment Effect, a.k.a. Effect Heterogeneity

Quicklink: http://bit.ly/2018ABTestingTutorial