How to avoid unreliable data

Traditional market research can produce distorted information, but brands are now using big data and new technology to better understand not only what people do but why they do it.

The Tube, TFL

TfL uses big data to predict demand for its services but small-sample qualitative data to understand why people travel in the way the do

Unreliable data at a glance

Two weeks before the Scottish referendum the polls had the Yes and No votes neck-and-neck, with one YouGov poll for the Sunday Times putting independence supporters ahead 51% to 49%. Political panic ensued, but by the time of the result on 19 September, a gap of 10 percentage points had emerged in favour of No.

It was a reminder that what people say they will do and what they actually do are often very different – something marketers know all too well. The bigger the gap between your research sample size and the size of the market, the greater the chance of disparity.

Yet marketers are finding ways to close this gap with research methods that create insights at scale. ITV has developed one such technique for the testing of TV content, which it calls ‘industrial qual’. The tool serves up real-time engagement metrics and thanks to offering viewers sliding scales and buttons such as ‘I would have switched off by now’ – reveals at what point in a show engagement peaks or drops off a cliff.

Elinor Bateman of ITV’s research and planning team, who will be discussing the industrial qual method at next month’s Festival of Marketing, says it provides data so decisions can be based on fact as well as hunch. What’s more, it can be turned around in 24 hours.

The research results derived from ITV’s bespoke online tool and LiveView app “have allowed us to get a much deeper understanding of our programmes both at pilot stage and also as they go out live on air,” Bateman explains.

“In-home dial testing at scale not only seeks to reduce the research effects of traditional methods but also allows us to analyse the data at depth, targeting certain demographics or viewer types.”

Top challenges

1. Be clear about what you are trying to discover. With data comes opportunity but also paralysis. “There are so many different places you can go to find answers that you can tie yourself in knots,” says RSA Insurance Group head of customer data and analytics Ian Kenealy. “Be clear about what you’re trying to discover. Market research is a very expensive but a very rich source of information so you want to maximise that opportunity.”

2. Be wary of research fatigue. It’s all too easy to set up a survey and gather masses of data, but will it reflect consumers’ attitudes? With online surveys there’s a temptation to overload customers with questions, provoking quick-fire responses rather than real insight. Channel 4 head of advertising research and development Martin Greenbank says the broadcaster gets “incredibly high” satisfaction scores from its research groups because the questions are interesting and respondents have an affinity with the brand.

3. There’s no silver bullet. All research techniques have their limitations. Simon Lloyd, head of analytics at Weve, explains: “Face-to-face questionnaires can influence the respondent into answering unnaturally; internet surveys don’t have the same gravitas and can be answered untruthfully; and behavioural data doesn’t have the depth of response [as to the] reason why someone behaves in a particular manner that marketers crave.”

The distortive effects of traditional methods are explained by Siamack Salari, visiting lecturer on insight generation at King’s College London: “It’s well known that traditional methods of research can result in unreliable data. The instant you ask a question you bias the response. The good news is that new methods of research can successfully capture people’s natural behaviour and emotions without the intrusive influence of a researcher.”

The days of using consumer surveys to garner a vague understanding of what people do and small focus groups to work out what makes them tick are not necessarily gone, but things are evolving.

“It can be a challenge in traditional market research to accurately depict real life and it is an issue the industry has been paying attention to in recent years by developing increasingly sophisticated methods,” says Hannah Bewley, senior research manager at the IAB.

Weve – the joint venture between the UK’s three largest mobile network operators EE, O2 and Vodafone – has been working with Havas Media on large-scale national studies. Head of analytics Simon Lloyd says his first instinct is to be “cautious of all results I’m presented with, regardless of their origin”.

“As a statistician, this caution grows further with traditional market research due to sample sizes and the age-old concern over ‘claimed behaviour’.”

Advances in internet and mobile technology have given researchers access to far greater pools of respondents, thus increasing sample sizes and their confidence in the results obtained. On the other hand, there is a fear that ease of response and anonymity could lead to a widening of the gap between actual and claimed behaviour (see ‘Top challenges’, below).

Simon Carter, executive director of marketing for Fujitsu UK & Ireland, says the general consumer distrust of business is harming brands’ insight efforts: “It is harder than ever to get a true answer from market research – many will give unduly negative comments as ‘payback’ and many others will simply refuse to engage.”

New research by the Confederation of British Industry shows only half of UK people believe that business makes a positive contribution to society, while two-thirds believe industry scandals (Libor fixing, PPI and, more recently, Tesco’s misstated financial results, for example) have had an adverse effect on their confidence in business.

There’s also a chance that consumers will refuse to take part in market research because it is boring – so-called ‘research fatigue’. It may be easier and cheaper to create and send dozens of questions in an online survey, but it can lead to quickfire responses that deliver little in the way of genuine insight. Focus groups, on the other hand, can provide invaluable behavioural insight, but are expensive.

Many companies, therefore, mesh the two. The transport industry, for example, has been using ‘big data’ for years – large-scale surveys whose main purpose is in modelling for transport planning in order to understand and predict demand. Where this data is less successful is in explaining why people do what they do.-

“That’s why we also use what I call ‘small data’,” says Ian Pring, research and insight manager at Transport for London. By this he means small-sample qualitative research; ethnography, where a few consumers give an in-depth record of their actions; and semiotics, where the symbolic power of branding is assessed. All this also feeds into TfL’s creative, produced by M&C Saatchi.

Pring explains: “People have a love-hate relationship with the transport system so one of the big challenges is to understand and account for these contradictory sets of attitudes – it’s very difficult with only one type of research to get to the heart of that.”

“We are trying to use transactional data to shape our research questioning”

Ian Kenealy, RSA Insurance

At Channel 4 the marketing team launched Core4 to understand what makes its viewers tick. There are around 11.5 million viewers registered on the platform, sharing opinions. But the broadcaster has gone one stage further with a sub-group of customers who are ‘on call’ to respond to questions about everything from programmes to advertising. The beauty of this model, says head of advertising research and development Martin Greenbank, is that they can match claimed behaviour with actual behaviour.

“It means we don’t have to ask them what they watched and when; we focus on the more interesting stuff such as their motivations.”

This ability to know the ‘what’ without speaking to the customer means Channel 4 can go straight to the ‘why’ when they do communicate – the important stuff for negotiating with advertisers.

Market research has got cheaper thanks to the web, but it is still an expensive outlay. For that reason it is critical to maximize the value of information, says RSA Insurance Group head of customer data and analytics Ian Kenealy. He admits it is “easy to tie yourself in knots”, and yet the alignment between transactional data and behavioural information is “much better than before”.

He explains: “We are trying to use transactional data to shape our market research questioning and we’re definitely seeing more combined insight projects where analytics and marketing are complementing rather than contradicting each other.”

Like Greenbank, Kenealy says the ability to know so much before even talking to a customer is a “big shift”. He uses the analogy of the recent Scottish referendum: “It’s the difference between asking people how they will vote, and getting very excited, and the ability to ask people who have already voted what the drivers [for their choice] were.”

Of course, you might not always like what you hear. Traditional engagement activity from companies often relies on talking with the people they know, the easy to reach, those most likely to give them what they want to hear. Keith Goffin, professor of innovation and new product development at Cranfield School of Management, says in this respect traditional market research “traps companies into developing incremental, me-too products”.

Market research is all about listening to customers. The way in which brands can do that is changing. Weve’s Lloyd sees data as “consumers talking to us directly; if we choose not to listen then we are breaking that bind of trust”.

He explains: “Sometimes as marketers, we forget that consumers don’t necessarily like or want advertising in the traditional sense. By listening to the consumer and targeting them with relevant ads, we can make advertising feel less pushy and more like something of value.”

Ian Pring, TFL

What are the limitations of traditional market research in representing consumer attitudes?

Traditional quantitative research doesn’t always get to the deeper psychological aspects of people’s attitudes, beliefs and behaviours. Even some ‘qual’ doesn’t get quite there, because there are some things that customers aren’t even aware that they are thinking and feeling, but nevertheless are.

Does this mean there’s a big gap between claimed and actual behaviour?

People are complex – they will do something knowing it’s not right and equally they won’t do the things they’d like to. In an area like cycling and walking, for example, it’s not about claimed versus actual, it’s about cognitive dissonance or a ‘knowing-doing gap’.

What research tactics does Transport for London use?

It’s useful to get people to talk about the behaviour but also to observe them. This was the principle behind the teen road safety research that won the Marketing Week Engage Award for insight in 2011. We conducted groups and observed teens’ behaviour on the roads. We found their peer group was a big motivator, hence the ‘Don’t let friendship die on the road’ messaging.

Latest from Marketing Week


Access Marketing Week’s wealth of insight, analysis and opinion that will help you do your job better.

Register and receive the best content from the only UK title 100% dedicated to serving marketers' needs.

We’ll ask you just a few questions about what you do and where you work. The more we know about our visitors, the better and more relevant content we can provide for them. And, yes, knowing our audience better helps us find commercial partners too. Don't worry, we won't share your information with other parties, unless you give us permission to do so.

Register now


Our award winning editorial team (PPA Digital Brand of the Year) ask the big questions about the biggest issues on everything from strategy through to execution to help you navigate the fast moving modern marketing landscape.


From the opportunities and challenges of emerging technology to the need for greater effectiveness, from the challenge of measurement to building a marketing team fit for the future, we are your guide.


Information, inspiration and advice from the marketing world and beyond that will help you develop as a marketer and as a leader.

Having problems?

Contact us on +44 (0)20 7292 3711 or email

If you are looking for our Jobs site, please click here