Home
Contact Us
Job Descriptions
Cool Things
Talks and Presentations
ExP Tools
What's a HiPPO?
Microsoft's Experimentation Platform
 

We have an unprecedented opportunity to run A/B tests with online users and innovate more quickly based on actual user response.
Microsoft needs to shift the culture from planning the exact features to planning a set of possible features, and letting customers guide us
   -- Ray Ozzie, Chief Software Architect         

Our Mission is to accelerate software innovation through trustworthy experimentation

Introduction

 

The Experimentation Platform enables product groups at Microsoft and later on will enable developers using Windows Live to innovate using controlled experiments with live users.   The platform enables testing new ideas quickly using the best-known scientific method for establishing causality between a feature and its effects: randomized experimental design.   The basic methodology in controlled experiments is to expose a percentage of users to a new treatment, measure the effect on metrics of interest, and run statistical tests to determine whether the differences are statistically significant, thus establishing causality.  The diagram below explains the concept:

 

 

Additional analysis against the collected data can provide more insights and enable testing other hypotheses.   Runtime monitoring, while the experiment is running, will enable early auto-aborts of experiments that are clearly underperforming, providing graceful failure (by auto-reverting everyone to Control).  The safety provided by this architecture encourages quicker deployments without extensive testing, and also allows for automated ramp-up from a tiny percent of users receiving treatment to a larger percentage since there are no adverse affects to the customer experience.

 

The platform encourages faster launches, which will in turn result in faster failures of bad ideas and better understanding of good ideas. 

 

  1. KDD 2009 Data Mining Case Studies Workshop, 3rd place winner: Online Experimentation at Microsoft (June 28, 2009)
  2. KDD 2009 tutorial: Planning, Running, and Analyzing Controlled Experiments on the Web (June 28, 2009).
  3. KDD 2009 paper: Seven Pitfalls to Avoid when Running Controlled Experiments on the Web (April 27, 2009).    
  4. CIKM 2008 industrial day invited talk and Video Oct 29, 2008
  5. Controlled Experiments on the Web: Survey and Practical Guide (February 2009, Vol 18, pp. 140-181)
  6. The amazing story of Semmelweis and the Semmelweis Reflex (Aug 24, 2008).  A 22-minute video about Semmelweis
  7. Bloodletting: Why Controlled Experiments are Important (May 19, 2008)
  8. Q&A with Mary Jo Foley (April 16, 2008).   (CNET news, Liveside, Germany, Japan, Poland, Turkey)
  9. Emetrics 2007 talk in Washington DC (October 2007): Practical Guide to Controlled Experiments (includes multiple examples).
  10. We published a paper that appeared in KDD 2007 (August 2007): Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO by Ronny Kohavi, Randy Henne, and Dan Sommerfield.
  11. A second paper in IEEE Computer (September 2007) is available: Online Experiments, Lessons Learned by Ronny Kohavi and Roger Longbotham (copyright IEEE, published with here with permission).

    12.  

     

    is live at Microsoft since 6/2007

     

    If you are a Microsoft employee, you can see more details at http://experiment

    If you are interested in learning more about job opportunities, see our job descriptions page.