Why Don’t Companies A/B Test Their Websites?

image of Dilbert cartoon about AB testing orange button
Source: Dilbert.com


Does your website have lots of obvious faults and usability issues? Are you fully aware of what changes need to be made to improve the performance of your site?  Do you lack some of the functionality that many of your competitors have on their websites? Have you employed a website design expert?

These are some of the many reasons I have come across why companies don’t conduct A/B testing on their websites. Despite the proven benefits of A/B and multivariate testing many websites continue to make changes without the use of online experiments.  Let’s examine the logic behind not testing.

Design Faults:


Amazon.co.uk homepage

Your website has lots of design faults and usability issues. Show me a website that doesn’t? There is no such thing as a website without usability and design issues.  Even Amazon and Google have issues with their websites and A/B test to understand how changes influence visitor behaviour. This is the nature of any choice architecture. Compromises have to be made when designing any website. The main unknown factor here is how these decisions influence visitor behaviour and conversion.



image of rainforest
Source: Freeimages.com

You can employ website design experts to recommend templates and changes to your website. They can employ best practice principles and obtain visitor feedback. However, given that every website is its own unique digital ecosystem it is impossible to predict how a change will impact user behaviour and your conversion goals. Measurement and testing should be employed to support this approach as otherwise you may not get the improvements in revenues that you anticipate.

An A/B test allows the impact of proposed changes to be measured because it splits traffic randomly and employs a control (the existing webpage) to identify the difference in the performance of the alternative user experience. This provides the confidence that any statistically significant difference in the conversion rate between the two experiences is  unlikely to be the result of changes in other factors (e.g. the quality of traffic, competitor activity or the weather) or just down random variance.

This scientific approach is essential for digital marketers as I have come across many instances where a clearly inferior design from a usability perspective results in a higher conversion rate. Stakeholders may still want to consider improving the user experience, but unless you run a controlled test you won’t be able to quantify the true costs or benefit of such a change.

Improved Functionality:


image of laptop and shopping cart
Source: Freeimages.com

Improving the functionality of your website unless relevant and implemented appropriately may not have the desired outcome. It may be a distraction or change visitor behaviour in a way that could damage conversion. In some cases A/B testing allows you to create new functionality without having to first build it internally. This means you can test and quantify the benefit against the cost of development before committing valuable and scarce resources to delivering something that may have limited or no impact on conversion.

Low Traffic:


image of a single car on a road
Source: Freeimages.com

Even if your site has low traffic or conversion rates there are still strategies that may allow you to benefit from A/B testing. In his article on how to test and improve your website if your traffic is too low for A/B testing  Rich Page suggest setting goals that take visitors towards your primary conversion objective, such as clicks on your call to action or engagement on the page.

I was asked to try to improve conversion on a mobile website with relatively low traffic. After reviewing the customer journeys I came up with two test ideas, one for new registrations and one for existing users. The first test increased first time deposits by around 50% and the second test improved revenues by 26% or over £500,000 a year. So don’t give up on your low traffic sites.


We Understand Our Customers:


Image of lady lying on the ground next to laptop
Source: Freeimages.com


Sure, you may have a good handle on who your customers are and why they come to your site. But human behaviour is too unpredictable to make changes to your website based purely upon subjective opinions and gut instinct. There is no such thing as a neutral choice architecture – every aspect of your website can influence visitor behaviour and not necessarily in the way you expect it to.

Expensive Risk Taking:


image of US $100 notes
Source: Freeimages.com

Adding content or making design changes to a website without testing is not optimisation, it is risk taking and it could cost your business millions in lost revenues. Although around half of my tests are successful I am frequently surprised by tests that don’t turn out as expected. Given this A/B testing is an essential tool for any digital marketer who wishes to understand how their content influences their prospects and customer behaviour.

Testing Culture:


image of the word Test
Source: Freeimages.com

For A/B testing to take off there needs to be an evidence based decision-making culture in the organisation. As people gain experience psychologists have shown that we become over-confident in our ability to predict outcomes. People also often copy their competitors as they assume they have based their website design on what works or they may have even A/B tested key pages themselves.  This is a risky strategy as many companies don’t A/B test and you end up with a website that looks like your competitors, thus reducing your ability to differentiate your brand.   Testing challenges the culture in many organisations that managers make decisions based upon their expertise and knowledge.

In his brilliant book The Wisdom of Crowds: Why the Many Are Smarter Than the Few, James Surowieckie  uncovers substantial evidence to show that individuals, small groups, and ‘experts’ in particular, are very poor at predictions and forecasts. Large, diverse and independent thinking crowds are much better at such decisions. That is why A/B testing is so powerful because it also harnesses the crowd (i.e. your visitors) to tell you what works best.

Thank you reading my post. If you found this useful please share with the social media icons on the page.

You can view my full Digital Marketing and Optimization Toolbox here.

To browse links to all my posts on one page please click here.

Recommended reading:


Landing Page Optimization: The Definitive Guide to Testing and Tuning for Conversions

  • About the author:  Neal provides digital optimisation consultancy services and has worked for  brands such as Deezer.comFoxybingo.com, Very.co.uk and partypoker.com.  He identifies areas for improvement using a combination of approaches including web analytics, heuristic analysis, customer journey mapping, usability testing, and Voice of Customer feedback.  By  aligning each stage of the customer journey  with the organisation’s business goals this helps to improve conversion rates and revenues significantly as almost all websites benefit from a review of customer touch points and user journeys.
  • Neal has had articles published on website optimisation on Usabilla.com  and as an ex-research and insight manager on the GreenBook Blog research website.  If you wish to contact Neal please send an email to neal.cole@outlook.com. You can follow Neal on Twitter @northresearch and view his LinkedIn profile.