Experimentation produces dream outcomes (part 1)
Today is the first part of a two part series on experimentation. This first part will be the product management version and part two will be experimentation for wellbeing (focusing on sleep).
I love experimentation. One of my principles for product and life is “everything is an experiment” because we’re always trying something, then adapting to information from trying something new and applying that to optimize what we’re working towards.
Let’s consider a recent experimentation case within our product team. We embarked on a journey to optimize our newsletter signup. We had this hypothesis that pop-ups disrupt the experience of reading a page and users blindly exit them. So we launched an A/B experiment with a new version of the signup that stuck to our content pages instead of a pop-up.
In a controlled experiment, we divided our audience into two groups. The first group encountered the original UI with the pop-up, while the second group was presented with an updated signup in the content of the page. We tracked user interactions and conversion rates. In the case of this experiment, the new UI showed a whopping 45% increase in sign-up conversions, which we considered a huge success.
Here’s what I love about experiments:
Simple changes can have outsized impact: our UI change helped thousands more jobseekers find our newsletter signup when they wanted to instead of blindly clicking out of it.
You gain confidence from informed decision making: because of the data our experiment collected, we confidently made a feature change without any hesitation about the impact because we could see the signals in our experiment data.
It’s inherently user-centric: experimentation forces you to collect data with end-user involvement, making your decision-making connected to real user behavior. The product evolves in tandem with the user's input, engendering trust and engagement.
Failure/learning is normalized: Failure is how you know where not to spend your time and that’s just as important as where to spend your time. On my team, we celebrate in retrospectives when we have a failed experiment that we learned from. Learning from and accepting failed experiments will make your product better as you focus on what is working. And I believe it helps your personal wellbeing to accept and learn from failures in your work.
Experimentation should be a fun part of the well-heralded Build-Measure-Learn cycle and it’s a process that I hope all product managers face and embrace, including the failures.
Bonus: below is my old experiment backlog from The Muse.
