“We have an unprecedented opportunity to run A/B tests with online users and innovate more quickly based on actual user response. Microsoft needs to shift the culture from planning the exact features to planning a set of possible features, and letting customers guide us.”
-- Ray Ozzie, Chief Software Architect, Microsoft

The Experimentation Platform enables product groups at Microsoft to innovate using controlled experiments with live users. The platform allows teams to test new ideas quickly using the best-known scientific method for establishing causality between a feature and its effects: randomized experimental design. The basic methodology in controlled experiments is to expose a percentage of users to a new treatment, measure the effect on metrics of interest, and run statistical tests to determine whether the differences are statistically significant, thus establishing causality. Additional analysis, including data mining, can provide insights beyond the standard hypothesis testing. Runtime monitoring, while the experiment is running, enables early shutdown of experiments that are underperforming. Wide adoption with the safety provided by this approach will speed innovation and improvement of Microsoft online services. Experimentation, when used properly, encourages more agile design and release cycles and can literally transform the way we produce software in the online space. There are many challenges in conducting controlled experiments on the web that we don’t face in the offline space that you can help solve.

We are looking for a researcher-analyst to work with clients to carry out and interpret valid experiments to make rapid improvements to their websites. The statistical issues range from planning experiments to have adequate power and to be insensitive to extraneous factors to identification of robots to data mining to extract the most information to assess main effects and interactions of test factors. The tests themselves range from simple Control/Treatment to MultiVariable tests (MVTs) with many factors and interactions of interest.

The client services we offer range from a self service model where the client plans, runs and interprets their experiment with little or no intervention from us to the full service model where we are involved in every aspect of the test conducting customized designs and challenging analyses. For all clients we offer training in planning and analysis of online experiments and web-based tools for setting up their experiments and viewing results.

Our team functions like a startup, giving you many opportunities to learn and develop and giving you responsibility to help define our future services to clients. You will be a part of a change in the way Microsoft develops software for the web – from long cycle times for release of a boxed product to frequent testing of ideas to change the many websites under the Microsoft umbrella. In addition, you will be helping us plan future versions of this experimentation service to non-Microsoft sites over the next few years.

Baseline qualifications:

• At least one year conducting statistical analyses or data mining
• An undergraduate degree or equivalent in a quantitative area (e.g. math, statistics, engineering)
• At least one year experience with SQL and relational databases

Desirable attributes:

• Knowledge of statistical theory and applications
• Experience with one or more statistical software packages
• Experience with design of experiments or multivariable testing
• Experience in conducting training for adult learners
• A BS or MS in Statistics, Data mining or related field

To learn more about the Experimentation Platform, visit http://exp-platform.com. For questions, please e-mail roger.longbotham at microsoft dot you know what.