Top Challenges from the first Practical Online Controlled Experiments Summit

Appears in SIGKDD Explorations June 2019: Volume 21, Issue 1.  PDF

Somit Gupta (Microsoft), Ronny Kohavi (Microsoft), Diane Tang (Google), Ya Xu (LinkedIn), Reid Andersen (Airbnb), Eytan Bakshy (Facebook), Niall Cardin (Google), Sumitha Chandran (Lyft), Nanyu Chen (LinkedIn), Dominic Coey (Facebook), Mike Curtis (Google), Alex Deng (Microsoft), Weitao Duan (LinkedIn), Peter Forbes (Netflix), Brian Frasca (Microsoft), Tommy Guy (Microsoft), Guido W. Imbens (Stanford), Guillaume Saint Jacques (LinkedIn), Pranav Kantawala (Google), Ilya Katsev (Yandex), Moshe Katzwer (Uber), Mikael Konutgan (Facebook), Elena Kunakova (Yandex), Minyong Lee (Airbnb), MJ Lee (Lyft), Joseph Liu (Twitter), James McQueen (Amazon), Amir Najmi (Google), Brent Smith (Amazon), Vivek Trehan (Uber), Lukas Vermeer (, Toby Walker (Microsoft), Jeffrey Wong (Netflix), Igor Yashkov (Yandex)


Online controlled experiments (OCEs), also known as A/B tests, have become ubiquitous in evaluating the impact of changes made to software products and services. While the concept of online controlled experiments is simple, there are many practical challenges in running OCEs at scale and encourage further academic and industrial exploration. To understand the top practical challenges in running OCEs at scale, representatives with experience in large-scale experimentation from thirteen different organizations (Airbnb, Amazon,, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, Yandex, and Stanford University)  were invited to the first Practical Online Controlled Experiments Summit. All thirteen organizations sent representatives. Together these organizations tested more than one hundred thousand experiment treatments last year. Thirty-four experts from these organizations participated in the summit in Sunnyvale, CA, USA on December 13-14, 2018.  

While there are papers from individual organizations on some of the challenges and pitfalls in running OCEs at scale, this is the first paper to provide the top challenges faced across the industry for running OCEs at scale and some common solutions.



  1. Paul Rouke: Fascinating insights in this in-depth report


author = {Somit Gupta and Ronny Kohavi and Diane Tang and Ya Xu and Reid Andersen and Eytan Bakshy and Niall Cardin and Sumitha Chandran and Nanyu Chen and Dominic Coey and Mike Curtis and Alex Deng and Weitao Duan and Peter Forbes and Brian Frasca and Tommy Guy and Guido W. Imbens and Guillaume Saint Jacques and Pranav Kantawala and Ilya Katsev and Moshe Katzwer and Mikael Konutgan and Elena Kunakova and Minyong Lee and MJ Lee and Joseph Liu and James McQueen and Amir Najmi and Brent Smith and Vivek Trehan and Lukas Vermeer and Toby Walker and Jeffrey Wong and Igor Yashkov},

title = {Top Challenges from the first Practical Online Controlled Experiments Summit},
journal = {SIGKDD Explorations},
issue_date = {June 2019},
volume = {21},
number = {1},
month = June,
year = {2019},
url = {},
publisher = {ACM}


Gupta, Somit, Ronny Kohavi, Diane Tang, Ya Xu etal. 2019. "Top Challenges from the first Practical Online Controlled Experiments Summit." 21 (1).

Download PDF

SIGKDD Explorations June 2019
SIGKDD Explorations June 2019


Quick link to this page: