Microsoft's Experimentation Platform

We have an unprecedented opportunity to run A/B tests with online users and innovate more quickly based on actual user response. Microsoft needs to shift the culture from planning the exact features to planning a set of possible features, and letting customers guide us.
   -- Ray Ozzie, Chief Software Architect    

The Experimentation Platform enables product groups at Microsoft and later on will enable developers using the Experimenation Platform to innovate using controlled experiments with live users. The platform enables testing new ideas quickly using the best-known scientific method for establishing causality between a feature and its effects: randomized experimental design. The basic methodology in controlled experiments is to expose a percentage of users to a new treatment, measure the effect on metrics of interest, and run statistical tests to determine whether the differences are statistically significant, thus establishing causality.  The diagram below explains the concept:

Additional analysis against the collected data can provide more insights and enable testing other hypotheses. Runtime monitoring, while the experiment is running, will enable early auto-aborts of experiments that are clearly underperforming, providing graceful failure (by auto-reverting everyone to Control). The safety provided by this architecture encourages quicker deployments without extensive testing, and also allows for automated ramp-up from a tiny percent of users receiving treatment to a larger percentage since there are no adverse effects to the customer experience.

The platform encourages faster launches, which will in turn result in faster failures of bad ideas and better understanding of good ideas. ExP has performed experiments on over 20 Microsoft properties and continues to integrate with new product and service providing innovation in the experimentation space.