By - Neal Cole

How Should You Prioritise Your A/B Test Ideas?

Before A/B testing it is essential to prioritise tests

Getting Maximum Return on Investment:

Before you begin A/B testing it is critical to ensure you prioritise A/B tests so that you achieve maximum benefit for your organisation. This means you must be careful to prioritise the pages and journeys most likely to have a large impact on your organisation’s goals and related key metrics. However, at the same time you also needs to take into account the difficulty  and time taken to develop your test ideas.

Before you even think about A/B test ideas you should review and fix your user experience by conducting a thorough evaluation of your digital experience. Otherwise it is likely that your A/B tests will fail because new designs will be overwhelmed by existing usability and user experience problems.

Below I’ve outlined a simple process that is designed to help you prioritise A/B test ideas to get maximum impact from your efforts. If you haven’t selected an A/B testing tool yet then you can read this guide on how to choose an A/B testing tool which summarises all the main solutions.

It’s important to have evidence behind your test ideas and develop a strong hypothesis. Otherwise we are prone to confirmation bias and only change the things we perceive are worth testing. Indeed, some experts believe this leads to a majority of winning A/B test results being an illusion. Preparation is very important with  A/B testing programmes as it heavily influences whether you are successful or not.

How to prioritise:

Begin by looking at your web analytics to identify where there is most opportunity to test.

1. Use data to identify top entry pages and view data at a template  level

2. Combine all pages that have the same template to identify traffic  levels for testing

Prioritisation framework (PIE):

  1. Potential – How much improvement can be made – how poor is the current page?
  2. Importance – How valuable is the traffic to the pages
  3. Ease – How complicated will the test be to implement on the page or template

1.Potential – Identify really bad pages:

Use your experience and best practice to identify pages that could most benefit from testing. Use the following metrics and tools to help guide you prioritise tests with most potential:

2.Important pages – What makes a page important?

Use your web analytics and your marketing spend to assess:

  • High traffic pages
  • Top entry pages
  • Pages with expensive visits are more important
  • Identify source of traffic  and  cost

3.Easy test pages – Consider technical implementation:

Tests that include the following are generally more complicated:

  • Site-wide elements like buttons, banners and  navigation bars
  • Alternative site templates
  • Dynamic content
  • Pages controlled by CMS or platform
  • Alternative flows – multiple pages
  • Pages with server-side validation or  interaction
  • Phone call tracking·
  • Multi-goal tracking
  • Experiments with multiple languages
  • Where multiple stakeholder opinions need to be satisfied
  • Consider organisational barriers – such as internal politics

But also remember that challenging tests can be most rewarding – particularly site-wide  templates.

4.  Prioritise pages – Score each page  from 1 (low)  to 5 (high) on each of the 3 criteria and position accordingly in a matrix.

Image of PIE A/B test prioritisation matrix

Prioritise according to their total ranking for all three criteria in a matrix like the example above. This should be circulated around teams involved in proposing test ideas so that they understand how tests are prioritised. To get buy-in from other teams it may be worthwhile asking some of the more important stakeholders to have an input into the prioritisation process.

This process is based upon the Widerfunnel approach which you can find in  the excellent book: You should test that by Chris Goward. The book is a must read for anyone wanting to create an optimisation programme.

Review your prioritised list of A/B test ideas on a regular basis to check that it still aligns with business goals and what you have learned from your A/B test results.

Conclusion:

Prioritisation is crucial to any conversion optimisation programme because resources are finite and often scarce. Getting time with designers or  developer resource can be challenging and so you don’t want to waste any resource you get with tests that won’t potentially have a good return on investment. The PIE framework is a simple and useful way of prioritising your ideas and yes there are more complicated methods available, but I like the fact that it is not difficult to implement.

Good luck with your A/B tests and let me know if you use the PIE framework or some other method of prioritising test ideas.

Thank you reading my post. If you found this useful please share with the social media icons on the page.

You can view my full Digital Marketing and Optimization Toolbox here.

To browse links to all my posts on one page please click here.

  • About the author:  Neal provides digital marketing optimisation consultancy services and has worked for  brands such as Deezer.com, Foxybingo.com, Very.co.uk, partypoker.com and Bgo.com. He uses a variety of techniques, including web analytics, personas, customer journey analysis and customer feedback to improve a website’s conversion rate.
Call Me Now!