ExP Articles

Articles

Experimentation at Microsoft is constantly evolving, as a way to give back to the experimentation field we have selected some of our articles to be availalble publicly.

Here is a list of those articles:

IEEE Computer 2010 article (to appear): Online Experiments: Practical Lessons (copyright IEEE, published with permission)
KDD 2009 Data Mining Case Studies Workshop, 3rd place winner: Online Experimentation at Microsoft (June 28, 2009)
KDD 2009 tutorial: Planning, Running, and Analyzing Controlled Experiments on the Web (June 28, 2009).
KDD 2009 paper: Seven Pitfalls to Avoid when Running Controlled Experiments on the Web (April 27, 2009).
Controlled Experiments on the Web: Survey and Practical Guide (February 2009, Vol 18, pp. 140-181)

The amazing story of Semmelweis and the Semmelweis Reflex (Aug 24, 2008). A 22-minute video about Semmelweis

Bloodletting: Why Controlled Experiments are Important (May 19, 2008)

Q&A with Mary Jo Foley (April 16, 2008). (CNET news, Liveside, Germany, Japan, Poland, Turkey)

Emetrics 2007 talk in Washington DC (October 2007): Practical Guide to Controlled Experiments (includes multiple examples).

We published a paper that appeared in KDD 2007 (August 2007): Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO by Ronny Kohavi, Randy Henne, and Dan Sommerfield.

A second paper in IEEE Computer (September 2007) is available: Online Experiments, Lessons Learned by Ronny Kohavi and Roger Longbotham (copyright IEEE, published here with permission).

Our Test Manager has a Blog. Read his thoughts on "The processes, people, and technology of producing quality software"

Microsoft's Experimentation Platform

Accelerating software innovation through trustworthy experimentation