ExP Talks (reverse chronological order)
- A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments: PowerPoint slides or PDF slides (8/18/2022) based on KDD paper
- Trustworthy Online Controlled Experiments: Q&A with Alexa Lab Fishbowl, slides (4/28/2021)
- Trustworthy Online Controlled Experiments and the Risks of Uncontrolled Observational Studies, AI Impact 2020 slides (2/4/2020)
(Similar: KDD 2019 Causal Discovery workshop invited talk slides)
- Top Challenges from the First Practical Online Controlled Experiments Summit, MIT CODE 2019 invited talk: slides.
- Performance: Each 0.1 Second Is Worth $18M For Bing, Convex online conference invited talk recording (8/5/2019).
- Trustworthy Online Controlled Experiments at Large Scale at Decisions Conference 2018: slides, recording
- Best Refuted Causal Claims from Observational Studies: MIT Code 2018 invited talk slides.
- Metric Pitfalls in Online Controlled Experiments: MIT Code 2017 invited talk: slides.
- A/B Testing at Scale Tutorial given at SIGIR 2017 and KDD 2017)
- Pitfalls in Online Controlled Experiments: MIT Code 2016 invited talk slides, talk cartoon, 20 minutes video + Q&A (10/15/2016).
- A/B Testing Pitfalls: PPT (3/30/2016) at ConversionXL Live (3/31/2016)
- Fun talk I did to data scientists at Microsoft: Intuition Busters: video (2/16/2016)
- Challenging Problems in Online Controlled Experiments: MIT Code 2015 invited talk slides and talk cartoon (10/17/2015)
- Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 years: KDD 2015 Keynote (8/11/2015)
- Lessons from Running Thousands of A/B Tests: MIT Code 2014 invited talk slides, talk cartoon (10/11/2014)
- Online Controlled Experiments: Introduction, Insights, and Humbling Statistics video and slides from Qcon (11/11/2013)
- ACM Recommender Systems industry keynote (9/12/2012)
- Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained
- KDD Conference, Aug 2012. See paper.
- Tutorial: Controlled Experiments on the Web: Planning, Running, and Analyzing, Advanced Research Techniques Forum (ART 2012) (previous version at KDD 2009)
- Online Controlled Experiments: Listening to the Customers, not to the HiPPO (4/2010). Video of talk at SD Forum
- Experimentation and Analytics - closing the loop from strategy to execution by Michal Lenchner
- Analytics revolution at Microsoft (in German) and Translated to English]
- Keynote at EC10: PPTX
- KDD 2007 talk: video
- Testing with Real USers, Better Software Conference, June 2010
- Online Experimentation at Microsoft, 2009
- PowerPoint, PDF
- Greg Linden wrote: ... In the barely viewable video of the talk, the action starts at the Q&A around 1:28:00. The presenters of the two talks, Googler Sandra Cheng and Microsoft's Ronny Kohavi, aggressively debate the importance of performance when running weblabs, with others chiming in as well. Oddly, it appears to be Microsoft, not Google, arguing for faster performance.
- Top 7 Testing Pitfalls, ‘Best of’ Which test Won's 2009 Webinars, Nov 18, 2009 [no longer available on their site]