Why You Shouldn’t Use focus Groups To Evaluate Your Home Page! Pitfalls of A/B testing (Part 2)

image of focus group viewing facility

I recently attended a conference on conversion optimisation. One of the speakers talked about how they had used focus groups to provide feedback on alternative designs for their new home page. One of the designs received much more favourable feedback than any of the other variants. This gave the client confidence that this design would win over their visitors. However, when they used A/B testing to measure the performance of each variant the preferred design completely bombed. They were relieved that they had decided to test the design before changing their home page.

The outcome of the test did not surprise me. How we articulate what we think about a user interface design is very different from how we are likely to behave in reality. When we browse the internet we are normally seeking to complete a task that takes us closer to achieving a personal goal. This is not a group experience.

Further, neuroscience indicates that most of our decisions are taken intuitively, often with little conscious awareness. We use behavioural short-cuts to minimize cognitive load and our emotions can over-ride our rational thought processes.

How often do you browse the internet with a group of people you have never met whilst being observed through a one way mirror? Website navigation and task completion is more often than not undertaken by an individual without any group interaction. This may be changing to an extent with mobile devices, but in most cases these are still people we know.

Context is critical in how we respond to any product or service. And yet focus groups are often conducted in a completely alien and artificial environment which does not match the reality of website browsing. Viewing facilities exasperate this problem as they create a laboratory atmosphere.

Some times it is suggested that focus groups are used before designs are drawn up to understand user’s wants, needs and likes. I would also caution against this as focus groups are problematic on a number of counts:

  • A single person can dominate the discussion and respondents will sometimes change their opinion to conform to the majority or because they mistake confidence for knowledge.
  • People over-analyse and rationalize the topic when we are looking for their emotional  and intuitive response to achieving a goal. People will say one thing but behave in a totally different way in reality. This is because people are extremely poor at predicting future behaviour as we are prone to the influence of short-term emotions and cognitive bias.
  • Presentation is critical. Despite explaining to people that a design is a mock-up or not fully complete you can guarantee that the stage of development of a design will influence how people respond to it.
  • People dislike uncertainty and when it does exist people look to the actions of others to guide them. ‘Group think’ may also set in if a group is too homogeneous and we seek a consensus of opinion rather than having a show of hands. This is particularly worrying as participants to focus groups are normally recruited on the basis of being aligned to agreed demographic characteristics.

As a result I would certainly avoid using focus groups for evaluating new website pages or journeys. All research methods have their limitations but focus groups are especially problematic and inappropriate in the context of website design.

Individual usability interviews can provide much more meaningful insights about how a website or web page works or not. Learning to listen and observe rather than asking direct questions is likely to provide more useful feedback on how to improve your site. There are also other online research tools that can provide relevant sources of insight. These include:

Eye tracking: AttentionWizard.com – Uses algorithms to predict what page elements visitors are likely to look at before the page goes live.

Usability: UserTesting.com and Loop11.com allows you to submit task that visitors should try to complete and provide feedback through visual recordings of users.

Feedback from usability & conversion experts: ConceptFeedback.com

Thank you reading my post. If you found this useful please share with the social media icons on the page.

You can view my full Digital Marketing and Optimization Toolbox here.

To browse links to all my posts on one page please click here.


Further reading:



Consumerology: The Truth about Consumers and the Psychology of Shopping (new revised edition, including a new preface from the author)


Website Optimization: An Hour a Day


Don’t Make Me Think!: A Common Sense Approach to Web Usability


  • About the author:  Neal provides digital optimisation consultancy services and has worked for  brands such as Deezer.comFoxybingo.com, Very.co.uk and partypoker.com.  He identifies areas for improvement using a combination of approaches including web analytics, heuristic analysis, customer journey mapping, usability testing, and Voice of Customer feedback.  By  aligning each stage of the customer journey  with the organisation’s business goals this helps to improve conversion rates and revenues significantly as almost all websites benefit from a review of customer touch points and user journeys.
  • Neal has had articles published on website optimisation on Usabilla.com  and as an ex-research and insight manager on the GreenBook Blog research website.  If you wish to contact Neal please send an email to neal.cole@outlook.com. You can follow Neal on Twitter @northresearch and view his LinkedIn profile.