|
|
Experimentation Platform Talks / Presentations Randy Henne, Principal Group Program Manager, Experimentation Platform EMEA, Microsoft Corporation
Talk at Emetrics 2007 in Washington DC Practical Guide to Controlled Experiments (includes multiple new examples). Ronny Kohavi, General Manager, Experimentation Platform, Microsoft Corporation
Talk at Ebay Research Labs 6/6/2007 Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO (Paper)
Ronny Kohavi, General Manager, Experimentation Platform, Microsoft Corporation
The web provides an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called randomized experiments (single-factor or factorial designs), A/B tests (and their generalizations), split tests, Control/Treatment tests, and parallel flights. Controlled experiments embody the best scientific design for establishing a causal relationship between changes and their influence on user-observable behavior. We will look at several real-world examples with significant return-on-investment (ROI), where it was clearly better to experiment and listen to the customers and not to the Highest Paid Person’s Opinion (HiPPO). We will review the important ingredients of running controlled experiments, and discuss their limitations (both technical and organizational). We describe common architectures for experimentation systems and analyze their advantages and disadvantages. Based on our extensive practical experience with multiple systems and organizations, we share key lessons that will help practitioners in running trustworthy controlled experiments. Based on paper accepted to KDD 2007, co-authored with Randy Henne and Dan Sommerfield.
ACM Data Mining SIG June 14, 2006 This was the first talk of the Data Mining SIG of the local Bay Area ACM chapter in Palo Alto, CA. Focus the Mining Beacon: Lessons and Challenges from the World of E-Commerce (PPT)
More information at the event site: http://sfbayacm.org/events/2006-06-14.php Ron Kohavi, General Manager, Experimentation Platform, Microsoft Corporation Electronic Commerce is now entering its second decade, with Amazon.com and eBay now in existence for ten years. With massive amounts of data, an actionable domain, and measurable ROI, multiple companies use data mining and knowledge discovery to understand their customers and improve interactions. The talk will cover important lessons and challenges using e-commerce examples across two dimensions: (i) business-level to technical, and (ii) the mining lifecycle from data collection, data warehouse construction, to discovery and deployment. The talk will include examples from real-world A/B tests and Simpson's paradox.
Approaches to Online MVTs
December 12, 2007
Roger Longbotham, Principal Statistician, Experimentation Platform, Microsoft Corporation
This position paper introduces two approaches to carrying out MVTs online that I believe are superior to the most common approach but are rarely if ever used. By publishing this position paper I hope to get feedback from other practitioners on agreements and disagreements with my arguments. I realize that some disagreements may come because of different priorities or technical challenges. It will be helpful to hear many perspectives and the reasons for the agreement/disagreement. Please email me at Roger.Longbotham@microsoft.com
|
|