Regression to the mean describes the phenomenon that when the first measurement of a variable is extreme (i.e. significantly different from the average) it is usually followed by a measurement much closer to the mean. Alternatively if the first observation is close to the mean, the second measurement it likely to be further away from the mean. This occurs whenever there is a less than perfect correlation between two variables (e.g. high value game players and size of deposit).
Further, extreme groups, such as an exceptionally high or low performing sales region or country, will regress to the mean over time. To measure performance of extreme groups it is necessary to compare against a control group which would be expected to improve by regression alone. Only by having a control can you tell whether a new experience (e.g. a new landing page) is influencing performance or whether it is due to regression to the mean.
It is not uncommon for A/B tests to start with an extreme measurement and then for this to gradually disappear as the variant regresses to the mean. Further, a common mistake with low performing sales regions or countries is to introduce a new incentive or landing page with the hope of improving conversion. However, this is often implemented without a control (i.e. an A/B test) and so how can we be confident that any rise in conversion is due to the intervention or just regression to the mean?
Regression to the mean is an important phenomenon that can easily be confused with correlation in experiments and conversion reporting. It should make us wary of stopping experiments too early as many tests will start with an abnormally high or low score but then regress to the mean.
It also shows the value running A/B tests to compare the performance of extreme groups so that we don’t fall into the trap of assuming that an improvement in conversion in a poor performing region or country is down to an intervention when it may just be due to regression to the mean.
Further, if we find that a winning variant from an A/B test does not deliver the expected uplift when it is implemented this could suggest that you’re A/B testing software is not selecting visitors at random. If this is the case then your test result could actually be down to regression to the mean rather than a sustainable uplift.
Conversion marketing – Glossary of Conversion Marketing.
Over 300 tools reviewed – Digital Marketing Toolbox.
A/B testing software – Which A/B testing tools should you choose?
Types of A/B tests – How to optimise your website’s performance using A/B testing.