Insights from Failed Product Experiments and Their Lessons
Written on
Understanding the Value of Non-Significant Results
When I transitioned into the role of a product analyst, I initially believed that product experiments mirrored the A/B tests I had conducted as a marketing data analyst. However, I quickly learned that even experiments lacking statistical significance can yield important insights. In this discussion, I will share instances of product A/B tests that did not meet expectations and the lessons derived from them.
Unexpected Outcomes and Hypotheses
We introduced a new meal plan feature with the expectation that it would enhance the trial-to-paid (TTP) conversion rate. A higher TTP conversion rate would suggest more users transitioning to paying subscribers, ultimately boosting revenue. The experiment involved two equally sized groups: one with access to the new feature and a control group without it.
Results Revealed
Surprisingly, the new feature did not lead to any improvement in the overall TTP conversion rate. This was unexpected, especially since this feature had been highly requested by users for years. To delve deeper, I segmented the test group to uncover potential reasons for the similar TTP rates before presenting my findings to the product manager.
I categorized the test group into segments based on their awareness of the plan feature. Group A comprised users who learned about the feature after beginning their trial, while Group B included those who were unaware of it. Additionally, I further divided Group A into those who started a plan and those who did not.
After analyzing these segments, it became apparent that Group B exhibited a TTP rate equal to the control group's 30%, as they were unaware of the plan feature's existence. In contrast, Group A users who did not start a plan showed a TTP rate of 35%, while those who did had a TTP rate of 25%. This discrepancy prompted further inquiry.
With the plan feature lasting 28 days, users often forgot about it, leading to a reduced impact on their decision-making, particularly as evidenced by the low completion rates. This understanding proved insightful for the product manager, highlighting the need for notifications to remind users about completing their plans. Such reminders could lead to higher completion rates and ultimately boost TTP rates.
Interestingly, users in Group A who were aware of the plans but did not participate exhibited a higher TTP rate. We speculated that this suggested a perceived value in the plans feature, which encouraged users to convert to paying subscribers even without direct engagement.
Key Takeaways
- Test results may not always be as straightforward as they seem. If unexpected outcomes arise, segmenting users by different actions can reveal significant differences in conversion rates, providing actionable insights.
- Be mindful of external factors that might influence test outcomes. In this case, we coordinated with product marketing to delay notifications about the plan feature until testing was complete, which could have affected results.
Unintentional Consequences of Product Changes
Hypothesis for Engagement
Previous analyses indicated that users who logged food entries after signing up for the app were more likely to return. To enhance user engagement, we modified the signup process to encourage food logging immediately after signup.
Results of the Engagement Test
The test was active for four months but failed to demonstrate any improvement in user engagement, leading to its termination. Concurrently, we noticed a decline in trial starts, which puzzled my team. The product manager proposed that the decline might be linked to the engagement test.
Upon reactivating the test, trial starts returned to previous levels. Initially, we attributed this increase to overall business growth rather than recognizing the impact of the test. It turned out that a change in the upgrade screen's position during the test inadvertently boosted trial starts. This insight highlighted the importance of strategically positioning upgrade messages.
Key Takeaways
- Tests can have unforeseen effects on other key performance indicators (KPIs) that may not be directly measured. In this instance, while engagement remained unchanged, trial starts experienced a significant boost.
- Ensure that control and test variants maintain consistent elements aside from the changes being evaluated.
- Stay informed about planned experiments, even those outside your immediate scope. Engaging with product managers about their roadmaps can reveal insights into ongoing initiatives.
Final Reflections
Although it was initially daunting, my experience as a product analyst has been incredibly enlightening regarding product A/B testing. Whether you're new to this field or contemplating a role as a product analyst, I hope these insights serve as a valuable resource for your testing journey. Thank you for reading!
You might also like…
This video discusses the lessons learned from the Museum of Failure, shedding light on how failures can provide critical insights into product development.
In this presentation, a leading engineer from Google explores the reasons behind product failures and offers strategies to ensure your product's success.