Years ago, in 2005, a Greek scientist published a fascinating article in PLoS Medicine in which he argued that most current published research ‘findings’ are false.
The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.
The interesting thing about this, as Alok Jha points out in a thoughtful Guardian piece, is that this comes as no surprise to professional scientists. Which only serves to highlight the intellectual and ideological chasm that divides the culture of journalism from the culture of scientific inquiry.
Delivering the Orwell lecture recently, Guardian editor Alan Rusbridger plainly stated what journalists should admit more often: that newspapers are full of errors. “It seems silly to pretend otherwise,” he said. “Journalism is an imperfect art – what Carl Bernstein likes to call the ‘best obtainable version of the truth’. And yet many newspapers do persist in pretending they are largely infallible.”
Yep. What’s truly weird is how reluctant journalists (and politicians) are to admit error. The minute a politician even hints that a rethink might be under way in government policy, hacks (most of whom have never run anything other than, occasionally, a bath) are jumping down his throat shouting “U Turn!” Outside the scientific mindset, writes Jha,
changes in direction are anathema to the world order. Journalists, politicians, business people and everyone else do not enjoy owning up to errors, because it chips away at their perceived authority. In politics, such change is called flip-flopping. Journalists hide behind the fig leaf of reader trust. (This has never made sense to me – why would your readers trust you more because you don’t acknowledge mistakes?)
Uncertainty, error and doubt are all confounding factors in whatever method you use to get at the truth. Acknowledging it and developing methods against it has been absorbed into scientific thinking – the most consistently successful method humans have developed to discover truth – and it seems churlish not to learn that lesson for the rest of life too.”
It’s possible that the Levenson Inquiry might recommend measures to compel journalists to admit to the margin of error in their reporting. But somehow I can’t see that chaning the prevailing mindset of the trade, once memorably expressed in the dictum: “Never apologise, and never explain”.
All of which brings to mind Keynes’s famous put-down of a journalist who complained that he had changed his position on monetary policy: “When the facts change, I change my mind. What do you do, sir?”
