A/B testing can use a Bayesian approach

Bayesian A/B Testing

Definition:

Bayesian A/B testing uses Bayesian statistical theory to determine the probability that a hypothesis (e.g. B is different from A) is true or false. Bayesian theory assigns a probability to a hypothesis, whilst in frequentist statistics a hypothesis is tested without being given a probability.

Further, with Bayesian A/B testing past knowledge of the phenomena is fed into a prior probability which is then updated to a posterior probability using the data from the experiment at hand. When the prior and the posterior are combined this allows us to come a conclusion about the result of the A/B test.

Bayesian statistics often generate results for parametric models that are close to the results produced by frequentist methods, but here are some advantages of the Bayesian approach:

  • It provides a logical way of combining prior knowledge of a similar situation within a robust statistical framework. This allows you to include past information about a parameter and calculate a prior distribution for future analysis. When a new A/B test is conducted for the same parameter, the previous posterior distribution can be used as a prior.
Image of informed prior distribution
Image Source: Informed Prior Distribution

 

  • It generates inferences that are conditional of the data and are precise, without reliance on asymptotic approximation. Small sample inference works in the same way as if we had a large sample.
  • The approach obeys the likelihood principle. If two separate samples generate proportional likelihood functions for ϴ, then all inferences about ϴ should be identical for both designs.
  • It generates interpretable statistics on the probability of a hypothesis.

The main disadvantages of Bayesian statistics are:

  • There is no agreed method for choosing a prior and it requires skill to translate subjective prior beliefs into a mathematically calculated prior. If not done correctly it could lead to misleading results.
  • The posterior distribution can be heavily influenced by the selection of the prior and the selection of the prior is a subjective process this can easily be challenged by other stakeholders.
  • Bayesian analysis can require a high level of computational resource, particularly in models with a large number of parameters.  Chris Stucchio provides a useful example of using Bayesian statistics for conversion in his post Analysing conversion rates with Bayes rules.

Conclusion:

Both VWO and Google Experiments use the Bayesian approach for A/B testing.  Bayesian analysis can often generate similar results to the frequentist approach for A/B testing and so we could consider the statistical method to be a secondary issue not worth a great deal of attention.

The main advantage of Bayesian statistics is the ability to include historical data and to select a prior distribution. The main danger though is the subjective nature of the selection process for the prior could be distraction from the test conclusion. However, I firmly believe that the key issue is whether the A/B testing tool you choose is appropriate and meets the needs of your organisation. The nature of the statistical engine is less of a concern.

Resources:

Conversion marketing – Glossary of Conversion Marketing.

Over 300 tools reviewed – Digital Marketing Toolbox.

A/B testing software – Which A/B testing tools should you choose?

Types of A/B tests – How to optimise your website’s performance using A/B testing.

Digital Marketing and Insight