Guide To Selecting Testing Software:
A/B and multivariate testing tools are essential for digital marketers as they enable you to deliver and measure the relative performance of different user experiences through robust online controlled experiments. Increasingly they also enable you to personalise your customer experience and discover new customer segments based upon behaviour rather than just demographics.
What is A/B Testing?
A/B testing allows you to run online controlled experiments to measure the difference in performance between an existing webpage design (e.g. your home page) and a change to the page or a totally new design or user journey (see split path test).
A/B testing tools randomly select visitors for each design and use robust statistical analysis to measure the performance between the control and the variant. If there is a statistical difference between the two experiences we can say with a high degree of confidence (normally 99%) that it is down to the design and not other factors (e.g. traffic source) that often influence the success metric.
Most A/B testing solutions also offer multivariate testing as an alternative. Multivariate tests allow you to test changing content in two or more locations within the same experiment. With traditional MVT software this does require more traffic and as a result the test will take longer to conclude. However, AI algorithms can conduct massively complex multivariate tests using evolutionary processes to maximise conversion without needing more traffic than traditional solutions.
Questions to consider:
Selecting the right A/B testing tool is key to the success of your optimisation strategy and I’ve summarised below over 20 tools that offer at least basic testing capabilities. When deciding on an A/B testing tool it’s worth considering these questions:
- What web analytics support do you have to provide insights for test ideas?
- Do you have access to usability testing and other user feedback tools to identify problems with your site?
- How easy it is to get internal IT development resource to code more complex tests?
- Do you have designers who can assist in creating new web pages and user journeys?
- What level of technical expertise does your team have to set up tests?
- How does your conversion rate compare to some of your competitors and what would be the value to your business of a 1% uplift in conversion?
- How much resource can you afford to dedicate to testing?
- What level of traffic do have on your site and therefore how long will it take to complete tests? Take an average weekly count of unique visitors over the last 12 months to estimate traffic levels.
- Who will manage the implementation of successful A/B tests?
Client-side or server-side?
The benefit of client-side software is that you can build simple tests using an easy to use visual editor (What you see is what you get – WYSIWYG). This means you can build and serve test experiences with little, if any, support from IT developers. Even marketers with no technical training can quickly become competent with using most visual editors. However, there may be potential IT Security issues as the new experience is delivered by calls to the software vendor’s server. See my client-side glossary post for more details.
Server-side tools offer you greater control on what is being displayed and for this reason any security concerns from internal stakeholders are much easier to resolve. You will require support from IT developers and can even build your own testing engine if you have the in-house expertise. See my server-side glossary post for more details.
Managed or Self-Service?
It is important to consider these questions before you proceed with choosing a tool because your answers will determine whether you will need a managed service, self-service or hybrid solution. A managed service contract is where the software solution’s team provides consultancy, builds the tests and analyses the results for you. A self-service contract means you build and analyse your own tests.
A managed service tends to be expensive, but because the software provider is building and analysing the tests for you it can give your testing programme a good kick-start. A self-service contract is great if you have the technical know-how and resource or if you have little budget. However, it can be difficult to build momentum when there is no testing culture already established in the organisation. A hybrid contract may be right for those organisations that need some level of consultancy and support to get started, but plan to develop the resource and expertise in-house over time.
Ideally you should have dedicated resource to get maximum ROI from your A/B testing software as it is a time consuming and expensive process. However, even if you can get a full-time testing manager you still need to consider what support they can be given with developing tests and implementing the successful ones. Testing is a collaborative process, one person cannot do it on their own.
Getting An Optimization Process In Place:
You also need to have a clearly defined process for website optimization. Without this you will struggle to maintain momentum or achieve the kind of consistent and sustainable uplifts that one would expect from a systematic approach to optimization. How to optimize your website’s performance explains 6 types of tests that you should be using to improve your conversion rate.
Perhaps surprisingly, Google Analytics, which is a free tool, was rated bottom of the pile by users. It does lack real-time reporting, but perhaps more importantly is that it does not offer a visual editor. However, this is likely to change with the recent launch of Google Optimize, the free version of Google 360, its enterprise testing and personalisation product.
What You See Is What You Get (WYSIWYG) visual editors are essential for client-side tools because they allow even the least technical of digital marketers to design and publish A/B tests on existing web pages within a matter of minutes. Most WYSIWYG editors now allow you to set up page re-directs (split-tests) so that you can test whole new pages rather than simply changing the colour of a button or text. A re-direct enables you to test an experience on a different URL against your existing page design.
The review was segmented by size of business. In the small business sector Unbounce and Visual Website Optimizer are the market leaders and received the highest user ratings. Monetate also achieved a high rating despite not being among the most used suppliers. Among medium sized businesses Optimizely is the market leader, though both Oracle Maxymiser and Monetate achieved marginally higher user ratings.
Mobile App Testing Capability:
For me having the ability to test native apps is one of the most important criteria. Apps are still relatively new and it would be dangerous and fool-hardy to assume that the user journeys they deliver are anywhere near optimal.
If your organisation has an app with a reasonable amount of traffic then you should certainly be considering integrating with a solution that allows you to test and personalise the in-app customer experience. Many providers now offer SDK solutions to test within native apps and so it is worth exploring this as an option when you decide to begin A/B testing.
As you will see from the providers summarised below there are many different kinds of tools on the market, and depending upon your situation and budget, there should be at least one that meets your needs. If you are going through the process of deciding on an A/B testing tool then I would strongly recommend that you download the TrustRadius Buyer’s guide to A/B testing.
29 A/B Testing Solutions:
1. A/Bingo: Is a Free Ruby on Rails A/B testing framework written as a plugin. It is an extraction from Bingo Card Creator, where it is used in production. It should work for most recent versions of both Rails 2.3 and 3.0.
3. A/B Test Master: A personalisation and A/B testing solution that uses machine learning to serve content according to browsing history, their social media channel and past behaviour. This analysis is used to display a customised version of your website that will help maximize the chance of conversion rate.
4. Adobe Target & Adobe Premium: A comprehensive enterprise A/B, multivariate testing and personalisation platform that works across all devices. Now part of the Adobe Cloud which offers full integration with Adobe Analytics and content management products. Includes visual editor for micro-tests.
Target is popular with e-commerce sites for its real-time automated self-learning personalisation engine. Target lets you automate the targeting process with its machine learning algorithm that continually and automatically makes associations with various events and differences between predicted and observed responses.
It allows you to use pre-set or customised rules, including those defined by visitor location, to target content to a specific audience based on real-time data. Target has an extensive range of out-of-the box targeting rules for you to choose from.
You can also use APIs to integrate Target with data from web analytics, CRM, partner and other off-site sources to enable further customised segmentation and targeting.
5. Change Again: Visual editor tool for A/B testing changes to existing pages. Allows for full integration with Google Analytics and other popular analytic tools.
6. Convert: Uses a visual editor for A/B, multivariate and split URL testing. Includes a WYSIWYG editor, Google Analytics integration, revenue and conversion tracking, behaviour and segmented targeting.
8. Google Analytics (Free): allows you to test up to 5 versions of a single page, each delivered to visitors via separate URLs. No visual editor and reporting is via email. However, Google Optimize, the free version of Google’s enterprise testing and personalisation product is due to be launched shortly. You can register for Optimize here.
Optimize 360 (currently in beta) is Google’s paid for enterprise A/B testing solution which is part of the Google Analytics 360 suite of integrated solutions. Optimize 360 includes the ability to target audiences identified in Google Analytics, testing based on data layer variables and a visual editor (WYSIWG) interface to bring it in line with other similar A/B testing tools. This fully integrated solution now puts Google in direct competition with other major solutions such as Adobe and Oracle Maxymiser.
9. Growth Giant: Uses Multi-Armed Bandit theory to run continuous A/B tests. It dynamically directs traffic to the best performing pages.
10. HP Optimost (Previously Autonomy): A managed service A/B and multivariate testing platform aimed at medium to large companies. This removes the need for IT support from your internal service providers. However, the interface is not as intuitive as other providers and there is no visual editor as yet.
11. KISSmetrics: Known for its highly flexible data reporting options, it excels at funnels with its extremely accurate funnel based-data, and can trace data to real people. Advanced data reporting requires a high level of analytical expertise.
12. Leanplum: A mobile personalised messaging, user experience optimization, A/B testing and analytics platform.
13. Marketizator: A simple, but flexible to use visual editor tool for A/B and multivariate testing. A tool Free for up to 10,000 tested page views.
14. Modesty: A simple A/B testing and event tracking framework for Ruby on Rails.
15. Monetate: An enterprise A/B, multivariate and personalisation platform with functionality for delivering totally new experiences to test. Integrates with common analytics packages and offers a visual editor for self-service basic testing implementation.
16. Oban International: Targeting international websites Oban International have wealth of experience with optimising sites at a cultural level. They focus on cultural A/B and multivariate testing to identify factors that significantly influence conversion rates for cross-cultural sites.
17. Optimizely: A very popular self-service solution that provides the ability to quickly A/B test changes to existing pages or redirect to a new page using their user friendly visual editor interface. The WYSIWYG editing tool allows for A/B and multivariate testing and offers real-time reporting. For more complex tests or technical support Optimizely have many authorised partners to provide advice and developer assistance.
They also offer an SDK solution for personalising and optimising mobile apps. Their predictive analytics and machine learning technology helps identify which user experiences are worth personalising by revealing high value audiences and identifying under- and over-performing segments.
18. Oracle Marketing Cloud (Maxymiser): A comprehensive enterprise A/B, multivariate testing and personalisation platform with the ability to build and test totally new pages without having to involve your developers. Maxymiser can split traffic randomly and display new experiences from its own server. Integrates with most analytics (e.g. Google Analytics & Omniture) and visual behaviour tools (e.g. Click Tale).
The self-service visual editor tool (Visual Campaign Builder) allows you to test changing elements on existing pages (e.g. headline, image or CTA changes) within a matter of minutes and now offers the ability to set up A/B redirect campaigns and multivariate tests. Also has an SDK solution for mobile app optimisation.
19. Personyze: A SaaS platform for real-time visitor segmentation and personalisation. Offers a suite of services designed to improve visitor engagement, retention and conversion. Offers audience-orientated A/B and multivariate testing.
20. Qubit: An enterprise analytics, segmentation, personalisation, A/B and multivariate testing platform. Includes a visual editor for changing existing page designs. Now has full mobile App testing capability.
21. Scenario from Convertize: Scenario is a new client-side A/B testing solution with a simple visual editor to quickly create and execute online experiments. To assist users in developing hypothesis to test Scenario have identified and developed over 200 tactics based upon neuroscience and consumer behaviour.
Simply select whether your site is an e-commerce, Saas or lead generation website and the software generates a menu of tactics to choose from. This means you don’t have to be a psychologist to develop persuasive ideas to test. Multivariate testing is not yet available, but is in the product roadmap for later in 2017.
Scenario offers new users a 14 day free trial period and a free plan after that expires for up to 5,000 visitors a month. However, a starter plan (for up to 50,000 visitors a month) costs just £29 per month. The Team plan for committed marketing teams (up to 300,000 visitors per month) costs £179 per month and the Agency plan (from 1,000,000 visitors per month) costs £499 per month.
22. Sentient Ascend: Ascend is the first pure artificial intelligence (AI) conversion rate optimisation software on the market to offer complex and fast multivariate testing. Ascend uses evolutionary algorithms to learn, adapt and respond to user interaction to identify the best performing combination of changes for your website. The AI technology needs less time and less traffic than traditional multivariate testing methods and by automating the testing schedule it speeds up the whole process from end-to-end.
Ascend is a client-side product which uses a visual editor to allow you to make changes. Ascend allows you to stop or adjust tests mid-way through to add new ideas based upon initial results. However, unlike traditional A/B testing algorithms, Ascend is always testing in the background and so if your traffic mix changes and a different experience would improve conversion, Ascend will automatically respond by serving the best performing design for that audience.
Sentient claims it can increase the speed of testing from around 10 to 100 times that of traditional testing software. Indeed, for underwear brand Cosabella Ascend tested 15 different changes to the homepage header, category page, product page and shopping cart design. Using standard multivariate testing would have required 160 tests, instead of the automated process that Ascend manages for you. This improved conversions by 35% compared to the control experience.
For more details of Ascend go to the post; How is AI disrupting conversion rate optimisation?
23. SiteSpect: An enterprise A/B, multivariate and personalisation platform for medium to large businesses. Includes a visual editor for micro-changes to existing pages.
25. Splitforce: A mobile A/B testing platform for iOS, Android and Unity Apps using a server-driven method. It randomly selects users on first app launch and saves dynamic objects onto the users’ device memory so that it can be accessed without a connection for every session thereafter.
26. Taplytics: This is an enterprise native app A/B testing solution that also offers push notifications with campaign automation to optimise both the timing and content of your communications. A visual editor allows for easy development and implementation of simple A/B and multivariate tests. More complex tests can be built by developers. Clients such as Tinder demonstrate that this tool has the capacity for scale and they offer a 30 day free trial.
27. Unbounce: The A/B testing tool for building, publishing and measuring the performance of landing pages using best practice templates provided by Unbounce. The solution also offers fully mobile responsive pages to A/B test.
28. Vanity: This is a Free “experiment driven development framework” for Ruby on Rails. It is an open-source program for A/B testing and is able to integrate with Google Analytics.
29. Visual Website Optimizer: Positions itself as the world’s easiest A/B testing tool. Provides a simple to use visual editor tool that offers A/B and multivariate testing capabilities, together with personalisation, heatmaps and user insights. Now offers an SDK solution for full mobile app optimisation.
VWO now uses bayesian statistical analysis which allows you to incorporate past experience when developing your tests.
With the advent of artificial intelligence optimisation tools are becoming increasingly powerful and responsive to the needs of digital marketers. However, it is important to take your time and consider the questions I raised above before choosing a solution.
These tools have great potential to improve your conversion rate and allow your optimisation team to become a revenue generator rather than a cost centre. But this will only happen if you follow a structured approach and create a culture of experimentation in your organisation. CRO is a collaborative and creative process that requires patience and resourcing appropriately.
Thank you for your interest in my post and hope you found it of help in your search for an A/B testing tool.
You can view my full Digital Marketing and Optimization Toolbox here.
To browse links to all my posts on one page please click here.
- About the author: Neal provides digital optimisation consultancy services and has worked for brands such as Deezer.com, Foxybingo.com, Very.co.uk, partypoker.com and Bgo.com.
- Neal has had articles published on website optimisation on Usabilla.com and as an ex-research and insight manager on the GreenBook Blog research website. If you wish to contact Neal please send an email to firstname.lastname@example.org. You can follow Neal on Twitter @northresearch, check out the Conversion Uplift Facebook page or connect on LinkedIn.