Reasons
Part of the principle of Lean Startup is to measure how customers respond, and we believe little changes can lead to big improvements. Hence the team has decided to use Google Content Experiments to gather data related to the customer response.
Steps
The first AB test implemented on BMJ Learning was a trivial test to find out whether minor changes to our homepage would attract more registrations.
Hypothesis:
Homepage with different colour button will have a higher conversion rate than a homepage page with the same blue colour button.
Success Metric:
Conversion rate as measured by the number of button click over the number of visits to each page.
Volume and Statistical Significance:
Test run for 38 days and accumulated 21,540 experiment sessions to the three pages.
Test Group and Split:
We set Google Content Experiments to distribute traffic evenly across the three variants of the homepage.
Randomization:
Google Content Experiments was used to run the test and make sure that the traffic driven to these pages was randomly distributed between the three test pages.
Winner
The winner of first AB test was the second version with a conversion rate of 4.55% compare to the original and the third version. This shows that the more obvious registration button will attract more attention.
The second AB test implemented on BMJ Learning was a test to find out if users prefer the registration journey on a single page.
Hypothesis:
One page registration journey will have a higher conversion rate compare to multiple pages.
Success Metric:
Conversion rate as measured by the number of users that completed the registration journey.
Volume and Statistical Significance:
Test run for 15 days and accumulated 20,059 experiment sessions to the two pages.
Test Group and Split:
We set Google Content Experiments to distribute traffic evenly across the two variants.
Randomization:
Google Content Experiments was used to run the test and make sure that the traffic driven to these pages was randomly distributed between the two test pages.
Winner
The winner of second AB test was the second version with a conversion rate of 6.34% compare to the original that only has 1.96%. This shows that a simple sign-up form attracts significantly more users than a more complex version.
Lessons learned
When there is no substantial data to back up certain changes and there isn’t enough market research to support a new feature, AB testing is a great way to find out the best solution for the customers instead of just relying on gut feelings or a figure plucked from the air.