Example of survivorship bias from US airplane in 2nd World War

Survivorship Bias

Definition:

Survivorship bias, or survival bias, is the tendency to focus on the people or things that managed to “survive” a process and so ignore those who did not succeed due to their lack of visibility. This can result in misleading conclusions being made because the nature of the process influences the characteristics of the survivors.

It can skew analysis and create over-optimistic attitudes as failures are not included in reviews. It may also lead to the false belief that successful members of a group have special characteristics rather than just coincidence (Correlation proves Causation).

Online Gambling:

For example, on gambling sites it is normal for a large proportion of revenues (e.g. over 50%) to be generated by a small number (around 5%) of players (often called VIPs). These players make large deposits and visit sites more often than lower value players. One might conclude that sites should focus on attracting these very high value players by offering high bonuses and high minimum deposit levels.

This would be a false conclusion because it ignores the process of becoming a VIP and how most VIPs either originate from lower value players or become one later on. Due to changes in circumstance and lifestyle people are continuously moving between different player segments, including VIPs. To analyse VIP players in isolation is folly.

VIP Gaming Sites:

Would it be sensible to establish a VIP only online brand for gambling? The problem here is that you would have to continuously siphon off your most profitable customers from your other brands and thus reduce their profitability. But you would also create the problem of what to do with customers who no longer meet the criteria for a VIP? Unless you migrate their account back to the original brand, which they might object to, you may have to continue to offer them benefits that they don’t qualify for. This just defeats the whole objective and reduces profitability.

Implications for conversion rate optimisation:

Survivorship bias has an important lesson for CRO – there is no such thing as a perfect experiment. We need to consider the journey the visitor has taken prior to conversion to identify whether they are a survivor and not to assume we can automatically apply learnings to other segments.

  • Test with new visitors rather than returning visitors:

Including returning visitors in your A/B tests is problematic because they may have already seen the old (default) content, but they are also unlikely to be representative of all prospects because most visitors don’t return to a site. Conversion rates also vary significantly by user type and each segment may not respond the same to interventions in your experiments.

  • Examine user personas:

Each user persona may have different intent levels due to the source of traffic or other factors driving their behaviour. For example direct traffic will often have much higher conversion rates than less qualified organic search traffic.

Conclusion:

It is easy to fall into the trap of survivorship bias if we don’t understand the different forces driving traffic to our sites. This is why it is important to interrogate data and dig deep into web analytics to identify how and why conversion rates vary so much by different user segments.

Resources:

Conversion marketing – Glossary of Conversion Marketing.

Over 300 tools reviewed – Digital Marketing Toolbox.

A/B testing software – Which A/B testing tools should you choose?

Types of A/B tests – How to optimise your website’s performance using A/B testing.

Digital Marketing and Insight