ExP Talks (reverse chronological order)

Pitfalls in Online Controlled Experiments: MIT Code 2016 invited talk slides, talk cartoon, 20 minutes video + Q&A (10/15/2016).
A/B Testing Pitfalls: PPT (3/30/2016) at ConversionXL Live (3/31/2016)
Fun talk I did to data scientists at Microsoft: Intuition Busters: video (2/16/2016)
Challenging Problems in Online Controlled Experiments: MIT Code 2015 invited talk slides and talk cartoon (10/17/2015)
Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 years: KDD 2015 Keynote (8/11/2015)
Lessons from Running Thousands of A/B Tests: MIT Code 2014 invited talk slides, talk cartoon (10/11/2014)
Online Controlled Experiments: Introduction, Insights, and Humbling Statistics video and slides from Qcon (11/11/2013)
ACM Recommender Systems industry keynote (9/12/2012)
Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained
  •  KDD Conference, Aug 2012. See paper.
Tutorial: Controlled Experiments on the Web: Planning, Running, and Analyzing, Advanced Research Techniques Forum (ART 2012) (previous version at KDD 2009)


Online Controlled Experiments: Listening to the Customers, not to the HiPPO


Testing with Real USers, Better Software Conference, June 2010
Online Experimentation at Microsoft, 2009
  • Video (46 minutes), PowerPoint, PDF
  • Greg Linden wrote: ... In the barely viewable video of the talk, the action starts at the Q&A around 1:28:00. The presenters of the two talks, Googler Sandra Cheng and Microsoft's Ronny Kohavi, aggressively debate the importance of performance when running weblabs, with others chiming in as well. Oddly, it appears to be Microsoft, not Google, arguing for faster performance.


Top 7 Testing Pitfalls, ‘Best of’ Which test Won's 2009 Webinars, Nov 18, 2009 [no longer available on their site]