# Welcome to A/B Testing at Scale Tutorial

Tutorial by: Somit Gupta, Ron Kohavi, Alex Deng, Paul Raff

Given at Strata San Jose 3/2018.

## Tutorial Outline

- Introduction to A/B testing (slides)
- What is A/B testing
- Brief history
- Why use A/B testing
- Examples
- Cultural Challenges

- Design of Experiments: Statistical Foundations (slides)
- Null-hypothesis testing, confidence intervals, p-values
- Central Limit Theorem
- Power analysis

- Ensuring Trustworthiness and High Quality (slides)
- Importance and example issues
- Data Quality
- A/A Tests
- Sample Ratio Mismatch tests
- Dealing with carry-over effects and random imbalance
- Twyman's law

- Importance and example issues
- Protecting Users (slides)
- Start small then ramp up
- Near-real-time detection and shut down of bad experiments
- Interaction prevention
- Interaction detection

- Designing Metrics (slides)
- Data Quality Metrics
- OEC (Overall Evaluation Criteria)
- Guardrail Metrics
- Local Feature and Diagnostic Metrics
- Metric Interpretation Pitfalls

- Active Research Areas and Recent Development (slides)
- Metric Sensitivity
- Issues with Null Hypothesis Testing and p-value
- Continuous Decision Making
- Beyond Average Treatment Effect, a.k.a. Effect Heterogeneity

Quicklink: http://bit.ly/2018ABTestingTutorial