Definition:

Bayesian A/B testing uses Bayesian statistical theory to determine the probability that a hypothesis (e.g. B is different from A) is true or false. Bayesian theory assigns a probability to a hypothesis, whilst in frequentist statistics a hypothesis is tested without being given a probability.

Further, with Bayesian A/B testing past knowledge of the phenomena is fed into a prior probability which is then updated to a posterior probability using the data from the experiment at hand. When the prior and the posterior are combined this allows us to come a conclusion about the result of the A/B test.

Bayesian statistics often generate results for parametric models that are close to the results produced by frequentist methods, but here are some advantages of the Bayesian approach:

  • It provides a logical way of combining prior knowledge of a similar situation within a robust statistical framework. This allows you to include past information about a parameter and calculate a prior distribution for future analysis. When a new A/B test is conducted for the same parameter, the previous posterior distribution can be used as a prior.
Image of informed prior distribution
Image Source: Informed Prior Distribution

 

  • It generates inferences that are conditional of the data and are precise, without reliance on asymptotic approximation. Small sample inference works in the same way as if we had a large sample.
  • The approach obeys the likelihood principle. If two separate samples generate proportional likelihood functions for ϴ, then all inferences about ϴ should be identical for both designs.
  • It generates interpretable statistics on the probability of a hypothesis.

The main disadvantages of Bayesian statistics are:

  • There is no agreed method for choosing a prior and it requires skill to translate subjective prior beliefs into a mathematically calculated prior. If not done correctly it could lead to misleading results.
  • The posterior distribution can be heavily influenced by the selection of the prior and the selection of the prior is a subjective process this can easily be challenged by other stakeholders.
  • Bayesian analysis can require a high level of computational resource, particularly in models with a large number of parameters.  Chris Stucchio provides a useful example of using Bayesian statistics for conversion in his post Analysing conversion rates with Bayes rules.

Conclusion:

Both VWO and Google Experiments use the Bayesian approach for A/B testing.  Bayesian analysis can often generate similar results to the frequentist approach for A/B testing and so we could consider the statistical method to be a secondary issue not worth a great deal of attention.

The main advantage of Bayesian statistics is the ability to include historical data and to select a prior distribution. The main danger though is the subjective nature of the selection process for the prior could be distraction from the test conclusion. However, I firmly believe that the key issue is whether the A/B testing tool you choose is appropriate and meets the needs of your organisation. The nature of the statistical engine is less of a concern.

Resources:

Conversion marketing – Glossary of Conversion Marketing.

Over 300 tools reviewed – Digital Marketing Toolbox.

A/B testing software – Which A/B testing tools should you choose?

Types of A/B tests – How to optimise your website’s performance using A/B testing.

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close