How Managers Can Build a Culture of Experimentation
“The hardest challenges in testing are internal processes, especially the need to work collaboratively to define a problem. Most managers are good at asking questions, but not as good (or, for various reasons, reluctant) at specifying what would constitute a feasible answer to those questions.”
Balancing Network Effects, Learning Effects, and Power in Experiments
“At DoorDash, we rely on experimentation to make decisions regarding model improvements and product changes because we cannot perfectly predict the results in advance. But experiments conducted by the Dasher team aimed at improving delivery quality and Dasher experience face challenges that arise from conflicting goals”
A/B Testing for Product Managers
“A/B testing (aka controlled experimentation) is used by top tech companies to build better product. I’ve built A/B testing at Disney Interactive and at Tinder, and the use of A/B testing continues to grow across the tech industry. This article is derived from a talk I gave to developing product managers at Product School in 2021, as a kind of practical introduction. Here’s why you should be A/B testing too!”
The Improvement of Our Randomization Algorithm
“Recently at Wish, we discovered a sample ratio mismatch (SRM) in an A/A test. SRM refers to
the mismatch between the sample ratio set by the experimenter and the observed sample ratio. SRM indicates potential data quality problems that change the outcomes of your experiments . Microsoft shared an example in which the A/B test results were flipped after resolving the SRM. But, how is it possible that SRM presents in an A/A test?”
Experimentation on news websites: How we differ from other businesses
“There are many factors at play of course, including industry type, objectives and team set-ups. For news organisations, there’s even more to consider. Here are three differences that I have found set experimentation at News UK apart from companies I have previously worked for.”
5 Reasons Why Experimentation Programs Fail
“Building and running a successful experimentation program is no easy task, and there are many twists and turns to it that can keep you from reaching your goals. Here are a few of the most common mistakes we see in experimentation programs, and how to overcome them.”