building an A/B testing plan using the test and learn approach

Iterative Testing Definition:

 

Iterative testing is a process of basing tests on insights gleaned from previous tests to make changes gradually and that are evidence based. Iterative testing is an important component of conversion rate optimisation. It allows a site to evolve over time with small steps rather than a large redesign. This has the advantage that users are often able to continue to navigate and browse without evening noticing many of the improvements made.

 

Example of iterative test which changed the social proof number of regular customers

 

After testing a new landing page against the home page we then decided to begin an iterative testing plan to understand the impact of individual changes on the conversion rate.  The above test changed the social proof below the call-to-action button. Just by changing the number from all players a month to the number of unique active players we measured a 5% drop in registration conversion. This confirmed the power of social proof and that we should test other forms of social validation.

1. Follow a systematic approach:

 

Iterative testing can provide more clarity about what changes improve or reduce the conversion rate. Making lots of random changes to the user experience can make it more difficult to understand what is driving the conversion rate. It also shows a lack of strategic direction because online experiments are most successful when there is a systematic process in place to create and develop a roadmap of tests based upon both qualitative and quantitative data.

 

2. Create a plan:

 

Creating a specific plan based upon previous tests enhances the learning process that A/B testing and multivariate testing provides to digital marketers. As each experiment is developed following a systematic evaluation of the previous test, marketers are less likely to fall into one of the many logical errors and cognitive biases that we are all prone to.

 

The Dunning-Kruger effect is one such bias where people misjudge the amount of expertise required to be competent at a task. This can lead to overconfidence and poor decision making. Working collaboratively in cross-functional teams can help avoid this problem and seeking peer reviews of your hypothesis.

 

3. Types of experiments:

 

A test and learn process is a much more scientific approach to website improvement than completely redesigning your website from scratch. Iterative testing is one of six kinds of tests that you should be considering using as part of a conversion rate optimisation programme. These include real estate tests where we move elements around to find their optimal position and inclusion and exclusion tests. Apart from innovation tests, these other kinds of tests may be included in iterative testing provided you have evidence to suggest they are appropriate to improve your key metric.

 

Iterative testing is one of six kinds of testing that are used in conversion optimisation

4. Should you redesign your site or app?

 

Redesigning your site is sometimes required when you have strong evidence to suggest that your site architecture and content is a barrier to further improving your conversion rate. However, iterative testing and other types of experiments should be attempted first. For example, Facebook did at one time regularly go down the route of a site redesign. What they found was that it resulted in a huge increase in complaints from users as they struggled to navigate and complete their common tasks.

 

People generally dislike  change and redesigns are problematic because users have to quickly re-learn how to navigate and browse the new site. Site redesigns will normally lead to a temporary drop in the conversion rate and some visitors may never return again if they find the new site too difficult to use.

 

For this reason be careful not to go down the route of a site redesign until you have exhausted all other options. Iterative testing and the other types of tests can achieve a great deal on most websites and apps. The most important aspect of all online experimentation is to ensure you follow a systematic process to evaluate the existing user experience and use data to develop strong hypothesis for testing.

Conclusion:

 

Iterative testing is a powerful means of improving your website or app by gradually evolving over time. This avoids disrupting and confusing users who dislike change and may abandon your site if you make lots of changes all at once. A site wide redesign should only be employed if all other forms of testing have been exhausted and you have evidence to suggest the current site has little potential for improvement.

 

Resources:

 

Optimize: How to set up and run tests in Google Optimize. 

A/B test plan: How to build a strong A/B testing plan.

Framework for optimisation. Conversion Rate Optimisation in 8 simple steps.

A/B testing software – Which A/B testing tools should you choose?

Types of A/B tests – How to optimise your website’s performance using A/B testing.

Mistakes in A/B tests – Are most A/B test results a lie?

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close