An A/B test is a controlled experiment where visitors are randomly selected to view one of two variations of the same web/app screen. Where the new design (often called the challenger) is on a separate URL it is known as split testing. Each design is referred to as a variant.
Here is an example of a successful split test for new visitors to the Cheekybingo.com home page. We created a simple splash page with a single prominent call-to-action and achieved a 27% uplift in registrations and a 9% uplift in first-time deposits.
As traffic is split randomly between variant A and B we can be confident that any statistically significant difference in the success metric (e.g. the conversion rate) between the two designs is likely to be due to the change in experience and not the result of other external factors (e.g. source of traffic, day of week etc). Most organisations use a specialist A/B testing tool to build, run and analyse their A/B tests.
An A/Bn test is where there are multiple variants of the same web/app page. For example an A/Bn test with 3 variants will mean that roughly 33% of visitors will see each variant. The more variants you test the smaller the proportion of visitors will see each variant and the longer the test will need to run to obtain a statistically significant result.
How should you use A/B tests?
A/B and multivariate testing should be used to validate changes to your website. Use a recognised approach to prioritise your test ideas to ensure you only test when it justifies the resource. A/B testing should be an important element of your website optimisation process. However, it is essential that you have a framework for optimisation that includes other inputs such as customer feedback, web analytics and usability research.
A/B and multivariate testing is not the same as conversion rate optimisation. It is one tool of many that should be employed as part of an optimisation strategy. However, it is the only way of scientifically validating what impact a change to your site has on your success metric.
Conversion marketing – Glossary of Conversion Marketing.
Over 300 tools reviewed – Digital Marketing Toolbox.
A/B testing software – Which A/B testing tools should you choose?
Types of A/B tests – How to optimise your website’s performance using A/B testing.