Scientific peer review: an ineffective and unworthy institution
By Paul Homewood
h/t stewgreen
The Times Higher Education Supplement’s blog carries a damning indictment of peer review, which has very real relevance to the climate change debate:
Given the entirely appropriate degree of respect that science has for data, the ongoing discussion of peer review is often surprisingly data-free and underlain by the implicit assumption that peer review – although in need of improvement – is indispensable.
The thing is, the peer review of scientific reports is not only without documented value in advancing the scientific enterprise but, in a manner that few care to acknowledge openly, primarily serves ends that are less than noble. Peer review is widely assumed to provide an imprimatur of scientific quality (and significance) for a publication, but this is clearly not the case.
While the many flaws of peer review are clearly laid out in the literature, its failure to protect the integrity of the scientific enterprise is notable. An estimated cost of irreproducible biomedical research is $28 billion (£20 billion) a year and “currently, many published research findings are false or exaggerated, and an estimated 85 per cent of research resources are wasted”, one paper found.
A prime example of the failure of peer review is the tainting of a significant segment of the biomedical literature by the use of misidentified and contaminated cell lines pointing, at best, to a culture of carelessness in cell biology research and the clear failure of peer review to discover and correct erroneous research.
There are many reasons why scientific peer review is ineffective. An important factor is the inadequacy of almost all scientific reporting; publications should contain sufficient information that all aspects of the work can be understood, permitting a published result to be reproduced from the original data, as well as independent replication of the study by others wishing to do this.
If these minimal standards are not met then critical information is missing and the reader has no way of assessing if the published research is correct or false in its claims and conclusions – even exact replication of a study is precluded. Reproducibility and repeatability require that all theory, methods, equipment, reagents, source code, computational environment, raw data and analytical and statistical methods be fully documented and openly available.
This standard is not enforced by peer review as currently practised, with the result that most publications in most journals should be viewed by the skeptical reader as little better than advertisements that present the authors’ claim to priority but preclude straightforward and independent verification.
We are not the first to identify these problems, so we might ask why peer review retains its essentially unassailable status. We suggest a two-fold answer rooted more in socio-economic factors than the dispassionate review of scientific research.
First, peer review is self-evidently useful in protecting established paradigms and disadvantaging challenges to entrenched scientific authority. Second, peer review, by controlling access to publication in the most prestigious journals helps to maintain the clearly recognised hierarchies of journals, of researchers, and of universities and research institutes. Peer reviewers should be experts in their field and will therefore have allegiances to leaders in their field and to their shared scientific consensus; conversely, there will be a natural hostility to challenges to the consensus, and peer reviewers have substantial power of influence (extending virtually to censorship) over publication in elite (and even not-so-elite) journals.
Publication in the highest-profile journals reinforces the hierarchies of status in the scientific community and promotes very effectively the prestige-, career- and profit-driven motives of authors, journal editors, publishers and (less directly) universities. This state of affairs exerts a particularly baleful influence on interdisciplinary research.
Innovations in peer review (including dispensing with its traditional forms) are to be encouraged. It may be that open publication through servers such as arXiv and bioRxiv, along with public and signed post-publication comment, are the solution to the problems noted above.
However, for any innovations in scientific publication to succeed two conditions would need to be met. The first, as noted above, is the provision with a publication of all the information necessary for independent reproduction and repeatability of the research, and the second is the improvement in the culture of science such that less than rigorous work and deceptive publication practices are no longer tolerated.
With the scientific method itself at risk, the stakes could not be higher.
Les Hatton is a mathematician and emeritus professor at Kingston University. Gregory Warr is a biochemist and emeritus professor at the Medical University of South Carolina. Both have extensive experience as peer reviewers and journal editors.
Comments are closed.
NOT A LOT OF PEOPLE KNOW THAT 
We are in a time when personal integrity is viewed as relatively unimportant in the scientific community at large. Thus institutions such as “peer review” fall prey to obvious pitfalls of people with no moral compass.
In the 1970’s, when working at the US National Herbarium of the Smithsonian Institution, I learned that many of the European scientific journals did not use peer review. The rationale was to allow the article to be published as written and for the author to stand for his/her work.
We certainly got a good look at the corruption embedded within “peer review” with the leaking of the UEA’s Climate Research Unit (Motley CRU as I call them) emails. They were seeking to have recalcitrant editors circumvented or even removed from journals and the un-cooperative journals ignored by their “scientific” community.
‘Motley CRU’ v. good, University of Easy Access as James Delingpole dubbed them.
Thou art not a man, thou’rt but a jester!
On with the motley, and the paint, and the powder!
The people pay thee, and want their laugh, you know!
If Harlequin thy Columbine has stolen, laugh Punchinello!
The world will cry, “Bravo!”
Pagliacci, Leoncavallo
As a former Editor, and associate editor, of more than one international scientific journal, I have seen it all. Much of the difficulty with “peer” review is embedded in the practice of allowing (even requiring) referees to maintain anonymity. These individuals can throw out unsubstantiated criticisms, (with aside innuendos to the Editor) and have no fear of accountability or of taking responsibility. It is especially pervasive with controversial topics. Forcing referees to sign their names would go a long way toward making the necessary review process fair and more workable. As a referee for many different scientific journals I always signed my name and wrote the author(s) independently so that they could respond to errors in my review that, if valid, I could acknowledge to the Editor as needed. I did that because at least one “high-profile” journal cuts off or blacks out any referee signature! No system is perfect but quality and fair reviews are necessary.
The problem is science itself and not peer review.
What is important is fact and science is historically the way we discover facts but today theory is accepted as fact. All of the so called evidence given for anthropogenic climate change is theoretical not factual and until and unless we go back to focusing on facts we will go on being robbed by our governments.
It’s scarcely even “theory”, just a heap of mathematical modelling, much of it performed unintelligently, uncritically, and even dishonestly.
“Peer review is widely assumed to provide an imprimatur of scientific quality”: and yet that was not the view when I was young and being instructed in my duty as a reviewer. I was told to ensure that the paper was clear enough, and complete enough, that the reader could make his own judgement on the merits of the work. In other words my enemy was muddle and obscurity, not error.
If I pointed to error that was just a useful by-product of careful reading.
rising to 100% of research resources wasted particularly concerning the field of rune reading mannramblingsof-inc and climatastrology.
In the field of climate science: “peer review” isn’t worth spit. coz, what else are your mates gonna say?
Yes, £20 billion wasted by irreproducible biomedical research but that’s dwarfed by the catastrophic effects of CAGW ‘research’. Nurse has a cheek puffing himself up with his ‘gotcha’ fatal disease question to Delinpole.
If in the future, a history is ever written, about how the west failed, it would reflect long on the UN agenda 21 treachery and not least focused on Britain and its unilateral suicide note.
The CCA, an act dictated to the British by its ‘own’ – legislature and otherwise known as the 2008 climate change perfidy.
Then ponder on, with useless input from our German fiends [no I didn’t forget the r] impositionss (and yes I did add an s) set via their Brussels proxies.
The breadth and scale of the great lie and ‘sustainability’ is the green evil, its waste and and deliberately done by the bureaucratic empire builders and the facile empty headed solutions to non existent problems, jobs galore for useless idiots and with their public money mugging agendas, is a monumental folly and egregious failure done by those who supposedly are, sworn to protect, secure and dedicate their efforts for the nation – and where in the actuality: they do all they can to sink us.
‘peer review’ is only but one part of it – the green disease.
I couldn’t agree more with the two authors. When I was gainfully employed and “getting it right first time” was the mantra, we used verification of any work before it was put out for review. The verifier had to be of equal or greater qualifications and experience than the author to carry out the role and he had to verify all the calculations and data etc, preferably by independent means. Of course there is no prestige or benefit to the reviewer in being a peer-reviewer for a journal, so it is mostly just a box-ticking exercise of very little, if any, value (except as we have seen in the field of climatology, to maintain the consensus and shut out any opposition to the settled “science”).
Reblogged this on Climate Collections.
Open review is far more useful and in this age of internet access quite easy to establish. Also I don’t see why public funded research papers should be locked behind a paywall. The complete 19th century advance in science was not peer reviewed.
No professional scientist accepts R D Cess’ claim that the ratio of OLR to theoretical S-B radiant exitance of Earth’s surface, if there were no atmosphere, is its radiant emissivity.
97% of climate scientists are, therefore, at lest unprofessional. Goody and Yung’s bidirectional photon diffusion model is based on forgetting that Planck 1913 assumed a vacuum. This is a story of monumental incompetence or deliberate fraud.
Over the years (since ’73, say) I’ve seen the following…and much more…
Limits to Growth – failed scare story
Global Cooling – failed as a child and disowned by its parents.
Ozone Hole – costly failure but a successful test of how science could be abused
Global Warming – dying failure – but very expensive – more-so in the effect its teaching has had on our children.
Diesel particulates – resurrected still-birth. Brought off the subs’ bench to keep the ball in play.
Plastics – genuine ecology problem that could be fixed at a fraction of the cost of AGW but being used as a diversion.
Alternative Green energy – potential West-killer but a dying failure hopefully.
Science corrupted – result.
All designed to scare the poor people of this planet and enrich the scammers. And make me the cynic I have become.
It was once the case that a researcher would follow a thread and come to a conclusion. Now they are told what the end game is and have a demand placed on them to find a suitable narrative in its support. We end up with the confident, strutting loquacious. We have an army of interlocutors masquerading as scientists all in a deadly embrace of money and misguided loyalty.
About 30 years ago, in a previous life, I did peer reviews for some fairly high-level publications and even then cronyism abounded. I agree that open reviews ought to be the norm: they were in the 1960s when letters, both critical and supportive of the paper, used to be published by academic journals. However, this was snuffed-out and the publications became more theoretical and less discussed. This encouraged the trait where authors write one paper then publish at least another 10 using the same material but re-presented to boost their credentials. Suspect any scientist with a mega list of publications!
Most journals will not publish critical letters now: that is a real disgrace to the integrity of once respected journals. It probably reflects the reduced competence of technical press editors who are more “journos” than technically competent writers.
At the root of much of this publication scandal is the “Publish or Perish” philosophy employed by so-called institutions of “higher learning.” Beginning in the 1960’s it has been widely employed, as a fund-raiser.
Basically, “professors” are hired on their ability to bring grant monies to the institution and thus pay for their own salaries by that method. Never mind that the institutions are largely publicly funded, but keep salaries of all except for their bloated administrations low to bring the steady flow of grant money.
Grants always come with “extra” for “overhead” which goes to the institution for their “handling” of the grant. At this time, overheads can run a whopping 60% above the grant itself and that is gravy for the “university.” Any wonder why we have publish or perish? There is a false, self-serving assumption that a top grant-getter is a great teacher. More often that is far from the case as many researchers are introverts who like to close themselves in their lab. But never mind, it is all about money and peer prestige and not the undergraduates.
I have watched as relatively cheap “organismal” botany and zoology programs are cut in favor of those cellular/molecular programs requiring very expensive equipment and labs. They bring in gobs of lovely grant money. Often medical schools are behind this ploy as doctors increasingly do not think anything not of their world is of the slightest importance. Administrations do not care that their biologists could not identify a plant or animal if they fell over them. Publish or perish has created a very unhealthy “dog-eat-dog” society in academia and peer review has become the absolute scandal we see.