How many scientific papers just aren’t true?
By Paul Homewood
h/t Philip Bratby
A very pertinent article by Donna Laframboise in the Spectator:
We’re continually assured that government policies are grounded in evidence, whether it’s an anti-bullying programme in Finland, an alcohol awareness initiative in Texas or climate change responses around the globe. Science itself, we’re told, is guiding our footsteps.
There’s just one problem: science is in deep trouble. Last year, Richard Horton, editor of the Lancet, referred to fears that ‘much of the scientific literature, perhaps half, may simply be untrue’ and that ‘science has taken a turn toward darkness.’
It’s a worrying thought. Government policies can’t be considered evidence-based if the evidence on which they depend hasn’t been independently verified, yet the vast majority of academic research is never put to this test. Instead, something called peer review takes place. When a research paper is submitted, journals invite a couple of people to evaluate it. Known as referees, these individuals recommend that the paper be published, modified, or rejected.
If it’s true that one gets what one pays for, let me point out that referees typically work for no payment. They lack both the time and the resources to perform anything other than a cursory overview. Nothing like an audit occurs. No one examines the raw data for accuracy or the computer code for errors. Peer review doesn’t guarantee that proper statistical analyses were employed, or that lab equipment was used properly. The peer review process itself is full of serious flaws, yet is treated as if it’s the handmaiden of objective truth.
And it shows. Referees at the most prestigious of journals have given the green light to research that was later found to be wholly fraudulent. Conversely, they’ve scoffed at work that went on to win Nobel prizes. Richard Smith, a former editor of the British Medical Journal, describes peer review as a roulette wheel, a lottery and a black box. He points out that an extensive body of research finds scant evidence that this vetting process accomplishes much at all. On the other hand, a mountain of scholarship has identified profound deficiencies.
We have known for some time about the random and arbitrary nature of peer reviewing. In 1982, 12 already published papers were assigned fictitious author and institution names before being resubmitted to the same journal 18 to 32 months later. The duplication was noticed in three instances, but the remaining nine papers underwent review by two referees each. Only one paper was deemed worthy of seeing the light of day the second time it was examined by the same journal that had already published it. Lack of originality wasn’t among the concerns raised by the second wave of referees.
A significant part of the problem is that anyone can start a scholarly journal and define peer review however they wish. No minimum standards apply and no enforcement mechanisms ensure that a journal’s publicly described policies are followed. Some editors admit to writing up fake reviews under cover of anonymity rather than going to the trouble of recruiting bona fide referees. Two years ago it emerged that 120 papers containing computer-generated gibberish had survived the peer review process of reputable publishers.
There are serious knock-on effects. Politicians and journalists have long found it convenient to regard peer-reviewed research as de facto sound science. Saying ‘Look at the studies!’ is a convenient way of avoiding argument. But Nature magazine has disclosed how, over a period of 18 months, a team of researchers attempted to correct dozens of substantial errors in nutrition and obesity research. Among these was the claim that the height change in a group of adults averaged nearly three inches (7 cm) over eight weeks.
The team reported that editors ‘seemed unprepared or ill-equipped to investigate, take action, or even respond’. In Kafkaesque fashion, after months of effort culminated in acknowledgement of a gaffe, journals then demanded that the team pay thousands of dollars before a letter calling attention to other people’s mistakes could be published.
Which brings us back to the matter of public policy. We’ve long been assured that reports produced by the UN’s Intergovernmental Panel on Climate Change (IPCC) are authoritative because they rely entirely on peer-reviewed scientific literature. A 2010 InterAcademy Council investigation found this claim to be false, but that’s another story. Even if all IPCC source material did meet this threshold, the fact that one academic journal — and there are 25,000 of them — conducted an unspecified and unregulated peer review ritual is no warranty that a paper isn’t total nonsense.
If half of scientific literature ‘may simply be untrue’, then might it be that some of the climate research cited by the IPCC is also untrue? Even raising this question is often seen as being anti-scientific. But science is never settled. The history of scientific progress is the history of one set of assumptions being disproven, and another taking its place. In 1915, Einstein’s theory of relativity undermined Newton’s understanding of the universe. But Einstein said he would not believe in his own theory of relativity until it had been empirically verified.
It was an approach which made quite an impression on the young Karl Popper. ‘Einstein was looking for crucial experiments whose agreement with his predictions would by no means establish his theory,’ he wrote later. ‘While a disagreement, as he was the first to stress, would show his theory tobe untenable. This, I felt, was the true scientific attitude.’
Real science invites refutation and never lays claim to having had the final say. As the US National Science Foundation recently pointed out, a scientific finding ‘cannot be regarded as an empirical fact’ unless it has been ‘independently verified’.
Peer review, as we have seen, does not perform that function. Until governments begin authenticating research prior to using it as the foundation for new laws and huge expenditures, don’t fall for the claim that policy X is evidence-based. It’s almost certainly not.
Donna Laframboise is the author of Peer Review: Why Scepticism is Essential, a report published by the Global Warming Policy Foundation.
Comments are closed.
NOT A LOT OF PEOPLE KNOW THAT 
For any given value of ‘true’?
Truish so long as you don’t look too hard?
A new Stern Review, this time on research: https://theconversation.com/stern-review-says-little-about-how-ref-has-affected-teaching-65853
“In the super-inflated market for star footballers, there is one thing a striker cannot do: move his winning goals to his new club. Not so in the almost equally inflated market for “star academics”. Here, researchers can transfer the credit for their publications to a new employer, at least for the purposes of the Research Assessment Exercise, the process of evaluating and ranking all university departments in the UK.
This is just one of the system’s absurdities that the long-awaited review by Lord Nicholas Stern wants to put right. The review, which was published over the summer, made a few well aimed recommendations to correct the most obvious anomalies of the Research Assessment Exercise, or REF.
The REF is a five or six-yearly evaluation of the research quality of each and every university department in the UK by panels of experts made up mostly of academics themselves. A great deal hangs on these evaluations, including institutional reputations, individual academics’ careers, student recruitment and opportunities for further research funding.
What the review refuses to notice is the extent to which the REF has turned academic research from a vocation to pursue knowledge and scholarship into a tyrannical game of “hits” in “top journals”. This has contributed to a massive growth in the numbers of research journals, with about 250 new ones starting every year. The number of published articles has also ballooned to over a million a year. Yet most of them languish unread and uncited.”
Is this a bit pot and kettle or poacher turned gamekeeper given Stern’s involvement in making false claims for published papers to get research cash?
http://retractionwatch.com
“Parkinson’s researcher Caroline Barwood was handed a two-year suspended jail sentence today after being found guilty of fraud yesterday by a jury in Brisbane, Australia.
A jury had found Barwood guilty of five out of the seven charges against her.
Earlier this year, Bruce Murdoch, a former colleague of Barwood’s at the University of Queensland (UQ) in Australia, pleaded guilty to 17-fraud-related charges, and earned himself the same sentence. In Barwood’s week-long trial, the court heard that she was previously in an intimate relationship with Murdoch. Both left the UQ in 2013.”
One fraud charge against Barwood — who was granted bail in 2014 — was related to obtaining a scholarship from the Lions Medical Research Foundation, McGuire said. Earlier this year, The Australian reported that the UQ had returned part of a $300,000 (AUD) grant to Barwood from the organization. The other charge, he added, was a result of including her name on two studies without justification.”
Sound familiar?
Why is the UEA “Motley CRU” not looking through bars about now? Why is Michael Mann still running around loose at Penn State?
Always one to watch http://retractionwatch.com/ , so much in medicine, psychology and gene manipulation is just junk.
Here is a good start with their weekend reads…
Agreed. It’s a very informative site.
Aficionados of science will remember the Phlogiston Hypothesis, created in 1667 by Johann Joachim Becher. This states that when a body is heated, ‘Phlogiston’ is driven out. It remained the dominant theory until the 1780s when Lavoisier proved that when substances are heated in air in closed flasks, they gain rather than lose mass. Thus previous experiments which allowed gaseous reaction products to escape, were invalidated.
A key supporter of this hypothesis was Joseph Priestley who, to get funding from the Classics’-educated aristocrats, persuaded them that Phlogiston was a 5th Greek Humour. So, Lavoisier’s proof that there was no 5th humour was a threat to his funding. The response was to claim Phlogiston had negative mass; ludicrous. Within 3 years, the theory was dumped and Priestley emigrated to America.
Climate Alchemy is following almost exactly the same evolution. It started with Arrhenius who assumed that our planet emits ‘black body’ IR which HEATS the local atmosphere. In turn, this HEATS the planet’s surface giving the GHE. The mechanism was developed by Simpson using Planck and Bose’s simultaneous bidirectional radiant heat transfer ideas, which only apply to a vacuum. Simpson’s real contribution was to point out that sufficient ‘atmospheric IR window’ means there can be no thermal runaway.
In 1960, Sagan and Pollack concocted an explanation of Venus’ hot surface as being caused by forward scattering of solar SW by clouds which then isotropically scattered IR thus reducing that planet’s atmospheric window, hence causing thermal runaway and decomposition of carbonates. Their incorrect physics which lives on in climate modelling by 1/3rd more low level cloud albedo than reality, hence imaginary positive feedback, made them the Godfathers of the climate scam, and glittering careers fleecing financial donors and in the case of Sagan, TV audiences (a bit like our Brian Cox).
Others followed: Cess made a bad mistake in 1976; GISS followed up with a fraudulent modelling paper which appears to have been written to push the CO2 is deadly scam to make the banks rich by carbon trading.
Our version of Priestley appears to be John Houghton, also highly religious, who lay preaches Thermageddon, presumably as God’s Punishment of Man. However, it convinced (for a while) Thatcher and got him to be chief of the Hadley Institute and co-founder of the IPCC, a political organisation intent on applying rejuvenated eugenics’ theory and making UN officials rich by fraud.
However, eventually real science catches up on the fraudsters. Thus there can be no mean heating of the local atmosphere by surface IR for the simple reason that for wavelengths outside the true atmospheric window, the IR emitting surface is many km above the surface. The GHE is really caused by the temperature gradient necessary to conduct surface SW warming mostly by convection, latent heat taken from the surface by clouds, then transformed to radiant emission to Space at various altitudes.
It’s much less than 33 K, but a mean GHE is also a ludicrous concept. What is real is minimisation of entropy production rate in the atmosphere (or possibly maximisation of potential energy) plus maximising radiation entropy production in OLR.The argument goes into a modified CLAW mechanism which is way more than any politician can cope with, or most academics who are too specialised to understand it. I give modern Phlogiston another 2 years at which time the key proponents blow fuses in various ways.
An excellent summation, thank you.
Peer review assumes the integrity of the authors and, at times, reviewers. It’s a system ripe for scams, but on the whole works pretty well.
“Pretty well”?
Pretty well as opposed to 100% certainty. The reviewers are not going to repeat the work, just judge that it has been done with reasonable methodology that could be reproduced by someone skilled in the art, the references are correct, etc.
Peer review works best when a referee is required to make his/her name known to the authors. Anonymity encourages referee bias. Manuscripts that “go-with-the-flow” are more likely to be recommended for publication than those that are skeptical or critical of dogma. Editors are not free from this bias either, especially when a topic is controversial. The referee who signs his/her name demonstrates integrity. And, if the author disagrees with the final editorial decision the source can be questioned directly.
I’m not sure how you can make that claim?
Lots of evidence shows that peer review is pretty appalling, a great deal of the time. Various teams are running simple algorithms over data for example and showing that significant numbers of papers have either basic arithmetic errors or have figures that cannot be right, or both. The implied error rate is around one third, and that is only the most basic problem, that a proper review should spot,
If peer review isn’t even catching this sort of stuff, how can anybody have confidence that it is catching other errors?
Peer review ceased to be based on integrity when scientists ceased to be men and women of integrity. It has all become politicized. We are at that place I predicted in the 1970’s where the general public no longer has trust in “science.” It will be a long road back.
When people claim peer review and science are not corruptable, that science is interested only in the truth, I ask if science is now being done by robots with impeccable, non-adjustable programming. If not, if humans are involved, there is no justification to the idea that science cannot be corrupted. Virtually everything human beings engage in can be corrupted—science, religion, politics, charity. To argue otherwise is a denial of human behaviour.
From my experience, professors and such ‘learned’ people lack objectivity – they never want to admit being wrong.
This was through showing how the Gini coefficient and other measures of inequality are statistical confidence tricks to hide worsening inequality – to protect the rich and powerful. Incomes always diverge in society and never converge – the poorest staying in poverty and the rich getting much richer.
Analogy: If the government measured weight problems by ignoring the obese and dangerously underweight – you would say they were corrupt and trying to hide the problems – wouldn’t you? That is how they measure inequality – ignore the richest and poorest groups – those millions of people most affected by what is being measured.
In your analogy, the problem isn’t that they ignore the obese and dangerously underweight in their analysis.
The problem is that they do not declare that that is what they have done.
It might be valid research, (marketing to those in the middle 🙂 ,) but the boundary conditions and assumptions need to be declared. It might highlight areas that are being ignored or even make the research pointless 🙂 but, without full declarations, it is likely to make the research non-repeatable, which is a mockery of Science, even if the conclusions are correct, for the (non-declared) cherry picked data!
I would have thought these assumptions would be an important part of the peer review – before the full experiments were conducted – it might even save some tax payers money!
Sounds like nonsense. Most of those who use the Gini coefficient want to show the opposite of what you claim, and generally do claim that incomes are become more unequal.
And since when does the Gini coefficient ignore the richest and the poorest?
Originally science was the thing that was going to kill religion, now science itself is a religion, complete with priesthood, fundamentalists (aka greens), inquisitions, and metaphorical witch and heretic hunts and burnings … all in the name of “saving the planet”.
A neat observation.
Ms. Laframboise does a very enlightening lecture here:
(It’s 31 minutes long, but, to her credit, she speaks so clearly that she’s perfectly understandable if you play the video at 1.5x or even 2x speed, which will reduce the playing time to as little as 15½ minutes.)
She has also written a highly acclaimed book.
Thanks for that link Dave. Shocking, that the legacy media haven’t picked this up.
Interesting and without a doubt true.
I have only reviewed books (for a prominent chemistry journal) so don’t know from personal experience what happens with journal publications
.
With the books – which always were related to statistical methods as applied to chemistry – my first approach was to seek out the numerical examples, type in the published data on which they were based (no downloads available) and carry out their reported analyses. I sometimes found clear errors, and before submitting my review I would contact the author, just in case I’d misunderstood something. These contacts were invariably fruitful. The author(s) accepted my findings, and I was usually able to point out what had gone wrong with their analyses. Incorrect data was typically the problem, and they were pleased to be able to correct this sort of mistake for the next edition.
So, perhaps some other reviewers do similar things – I don’t know – but in climate affairs I really doubt it!
Robin… Integrity. You signed your name to your review sent to the author. If everyone did that the peer-review system would surely improve. Some journals do not want the referee to become known to the authors and will hide or cross out any signature. My personal solution to that was to always write independently to the authors and tell them that I was a referee and if they had a problem with my review, contact me directly. It worked. Some were grateful, others were not…especially if their paper was rejected.
‘How many scientific papers just aren’t true?’
The question is a bit of a red herring. For example we often read ‘study suggests’ or similar, meaning even the authors know their ideas may not be correct, or are to some extent speculative.
Agreed some peer reviews are better, or worse, than others.
These two were linked by Dr. Judith Curry on one of her “week in review” blogs.
Why Most Published Research Findings Are False http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
Science isn’t self-correcting, it’s self-destructing. To save the enterprise, scientists must come out of the lab and into the real world. http://www.thenewatlantis.com/publications/saving-science
We could start with the fact that the theory of the greenhouse effect has yet to be proven either way.
The ‘bidirectional photon diffusion’ basis of the Arrhenius GHE mechanism is demonstrably false, theoretically and experimentally.
1. The theory is based on Planck and Bose’s analysis of radiant energy flow between emitters in equilibrium in a vacuum: the atmosphere is not a vacuum so the thermodynamic and quantum statistical theories of these two workers cannot be applied. there is on average zero heating of the local atmosphere by a planet’s surface.
2. The pyrgeometer instrument measures a potential, not a real energy flux.
Climate Alchemists desperately protect thee two fake scientific claims, even falsifying past data – about 22 years so far. This is the biggest Scientific and Economic fraud in History, a Mafia-controlled scientific and political establishment out of control. The political assassination of Pielke Jr by George Soros funded attacks is just the tip of the iceberg.
Ancil Keys’ Seven out of Twenty Two Countries Study still affecting government policy and advice 60 years after publication. Not necessarily improving people’s health.
Reblogged this on WeatherAction News.
I’m surprised she was so kind to the corrupt and fraudulent IPCC
“When a research paper is submitted, journals invite a couple of people to evaluate it. Known as referees, these individuals recommend that the paper be published, modified, or rejected.”
Actually a lot worse. This at least suggests that referees are basically honest, if under resourced and inept. In reality, as Climategate illustrated all too clearly, they are dishonest, over resourced and inept.
In short, pal review not peer review.
RW
Peer review means you get some friends who agree with you to read your research paper and tell you they agree. Rather than promote scientific curiosity and intellectual thought peer review actually leads to group think. As mentioned in other comments the result is junk science like the global warming hoax.
Yes, yes! True, true!
But what about the important stuff?
You know….Kim Kardashian’s arse….tonight’s hockey games…world series….