As Victor Meldrew would say: “I don’t believe it!” Research, that is. Almost any research, in fact. Examine any statistic quoted in the media at the moment and it’s as likely to be wrong as it is to be right. Sometimes by factors of a hundred, a thousand or more.
This, of course, is a wild generalisation, of the sort I am criticising, but it contains a good deal of truth. It is as valid as much of the research we are asked to believe and on which highly important decisions – sometimes crucial and life-critical decisions – are based.
The “I don’t believe it” factor is now seeping into the public’s mind, weakening the credibility of everything that bears the label “research” – market research, medical research, scientific research, media research.
Take Saturday’s papers as a snapshot. They reported that the Department of Health had apologised after one of its ministers told Parliament that 900,000 people claiming incapacity benefit were obese. It turned out that the real figure was 900. The minister had said that over &£70m a week was being paid to such people in England alone. In fact it was &£70,965. The department blamed an “administrative error”.
The same day, the editor of the Lancet announced he had been wrong to publish research that suggested a link between the MMR vaccine and autism because the doctor who wrote it had a potential conflict of interests. The General Medical Council has been asked to investigate, reopening the whole controversial issue.
The same day, the Daily Mirror reported that the sparrow population was being wiped out and the Government was blaming “trendy garden makeovers” for this. Figures from the British Trust for Ornithology (BTO) apparently showed that the number of sparrows breeding in Britain had almost halved since the Seventies.
In London and the South-east, according to the Mirror, the figure had fallen by 70 per cent in 30 years. Yet just as I was reading this, I heard the man from the BTO on Radio 4’s Today programme, saying the 70 per cent fall had happened in ten years – a far more dramatic decline.
So, 30, ten? Who’s counting? Indeed, who IS counting? I went to the BTO website to find out more. There was an impressive graph showing a steep decline from the Seventies to 2000. But that was from a different survey – the Garden Bird Feeding Survey. The new figures seemed to be from the BTO’s House Sparrow Questionnaire, filled in by more than 11,000 gardeners.
Buried deep in the site was a table headed “Facts and figures”. According to this, “House Sparrow populations have declined most in eastern England (90 per cent since 1970) and south-east England (65 per cent)”. Yet just below it was another table headed “Regional differences”, based on yet another survey – the Breeding Bird Survey, conducted between 1994 and 2002. According to this table, house sparrows really have declined in London by 71 per cent in the past ten years, as the man on the Today programme said.
The corollary is that in other parts of the country they’re thriving – up by 63 per cent in Wales. Is there a “sparrow drain” along the M4 corridor, with the birds deserting the city for a simpler, more rural existence? Or is it that there are now more birdwatchers filling in questionnaires in Wales and available to spot sparrows?
The figures may be confusing – but at least the basic finding of a decline in the sparrow population seems robust, even if I couldn’t find any evidence to support the minister’s claim that “the trendy makeover of gardens” was a key factor. The same can’t be said for much other research that gets headline treatment in the media.
Remember The Guardian’s front-page story “Audience with no one: 55 days on TV”? This said that more than 55 days’ worth of programmes on the BBC’s digital channels last year were watched by so few viewers that they scored zero on the BARB ratings.
The piece stated: “The channel with the most zero-rated output was BBC News 24, which recorded 472 hours of ‘unwatched’ programming, although much of it was in the early hours of the morning.”
What it didn’t explain was that in the early hours of the morning BBC News 24 is simulcast on BBC1 and that BARB allocates all such viewing to the larger channel. In fact, News 24’s programmes were watched by hundreds of thousands of viewers at that time. Buried deep in the piece was a statement from the BBC pointing out that the exercise had been carried out by one of its rivals and branding it as “spurious”. But the “55 days with no one watching” line was picked up by other newspapers and will continue to be trotted out by critics of the BBC’s spending on digital channels.
Research organisations already put health warnings on their surveys. These are routinely ignored. Yet they live and die by their credibility. If they allow themselves to become part of the spin business, more and more research will be greeted with the words “I don’t believe it”.