Peer Review – What it actually means 2

Here is more on the subject of the unscientific ways of peer review from Professor Briggs.

”that peer review ”is a non-validated charade whose processes generate results little better than does chance.”

”It is easy to get a paper into print when the subject is ”hot”, or when you are friends with the editor or he owes you a favor, or your findings shame the editor’s enemies, or through a mistake, or by laziness of the referees”

”Only 8% members of the Scientific Research Society agreed that ”peer review works well as it is”. (Chubin and Hackett, 1990; p.192).

”A recent U.S. Supreme Court decision and an analysis of the peer review system substantiate complaints about this fundamental aspect of scientific research.” (Horrobin, 2001)”

”Peer Review is central to the organization of modern science…why not apply scientific [and engineering] methods to the peer review process” (Horrobin, 2001).”

”(5) Then there is the Sokal Hoax, where a physicist sent a paper full of gibberish to a preeminent social science journal to see if it would be published. It was. Sokal was careful to play to the preconceptions of the journals’ editors to gain acceptance. The lesson is the oldest: people-even scientists!-easily believe what they want to.”

”The authors liken acceptance of papers in journals to winning bids in auctions: sometimes the winner pays too much and the results aren’t worth as much as everybody thinks.”

See also my posts:

Peer Review – What it actually means

Assessment of the reliability of climate predictions based on comparisons with historical time series

IPCC Review Editors – ”No Working Papers”, ”No Correspondence” are kept!,

The UN Climate Change Numbers Hoax eller IPCC:s lögn!

The Unscientific way of IPCC:s forecasts eller IPPC:s lögn del 2!,

IPCC Review Editors comments reveald!,

Has the IPCC inflated the feedback factor?,

IPCC and its bias! 

Article here:

http://wmbriggs.com/blog/2009/02/25/peer-review/

 Peer review

 Published by Briggs at 7:07 am

Here is how peer review roughly works.

An author sends a paper to a journal. An editor nearly always sends the paper on to two or more referees. The referees read the paper with varying degrees of closeness, and then send a written recommendation to the editor saying ”publish” or ”do not publish.” The editor can either accept or ignore the referees’ recommendations.

The paper is then either published, or sent back to the author for revisions or rejection.

If the paper is rejected, the author will usually submit it to another journal, where the peer review process begins anew. This cycle continues until either the paper is published somewhere (the most typical outcome) or the author tires and quits.

Here are two false statements:

(A) All peer-reviewed, published papers are correct in their findings.

(B) All papers that have been rejected1 by peer review are incorrect in their findings.

These statements are also false if you add ”in/by the most prestigious journals” to them. (A) and (B) are false in every field, too, including, of course, climatology.

A climatology activist might argue, ”Given what I know about science, this peer-reviewed paper contains correct findings.” This is not a valid argument because (A) is true: the climatology paper might have findings which are false.

If the activist instead argued, ”Given what I know, this peer-reviewed paper probably contains correct findings” he will have come to a rational, inductive conclusion.

But a working climatologist (gastroenterologist, chemist, etc., etc.) will most likely argue, ”Given my experience, this peer-reviewed paper has a non-zero chance to contain correct findings.” Which is nothing more than a restatement of (A).

The ”non-zero chance” will be modified to suit his knowledge of the journal and the authors of the paper. For some papers, the chance of correct findings will be judged high, but for most papers, the chance of correct findings will be judged middling, and for a few it will be judged low as a worm’s belly.

Here is a sampling of evidence for that claim.

(1) Rothwell and Martyn (abstract and paper) examined referees’ reports from a prominent neuroscience journal and found that referee agreement was about 50%. That is, there is no consensus in neurology.

(2) No formal study (that I am aware of) has done the same for climatology, but personal experience suggests it is similar there. That is, there is at least one published paper on which the referees do not agree (at what is considered the best journal, Journal of Climate).

(3) Pharmacologist David Horrobin has written a commentary on peer-review in which he argues that the process has actually slowed down research in some fields. He also agrees with my summary:

Peer review is central to the organization of modern science. The peer-review process for submitted manuscripts is a crucial determinant of what sees the light of day in a particular journal. Fortunately, it is less effective in blocking publication completely; there are so many journals that most even modestly competent studies will be published provided that the authors are determined enough. The publication might not be in a prestigious journal, but at least it will get into print.

(4) I have just received an email ”Invitation to a Symposium on Peer Reviewing” which, in part, reads:

Only 8% members of the Scientific Research Society agreed that ”peer review works well as it is”. (Chubin and Hackett, 1990; p.192).

”A recent U.S. Supreme Court decision and an analysis of the peer review system substantiate complaints about this fundamental aspect of scientific research.” (Horrobin, 2001)

 Horrobin concludes that peer review ”is a non-validated charade whose processes generate results little better than does chance.” (Horrobin, 2001). This has been statistically proven and reported by an increasing number of journal editors.

But, ”Peer Review is one of the sacred pillars of the scientific edifice” (Goodstein, 2000), it is a necessary condition in quality assurance for Scientific/Engineering publications, and ”Peer Review is central to the organization of modern science…why not apply scientific [and engineering] methods to the peer review process” (Horrobin, 2001).

This is the purpose of the International Symposium on Peer Reviewing: ISPR (http://www.ICTconfer.org/ispr) being organized in the context of The 3rd International Conference on Knowledge Generation, Communication and Management: KGCM 2009 (http://www.ICTconfer.org/kgcm), which will be held on July 10-13, 2009, in Orlando, Florida, USA.

Be sure to visit the first link for more information.

 (5) Then there is the Sokal Hoax, where a physicist sent a paper full of gibberish to a preeminent social science journal to see if it would be published. It was. Sokal was careful to play to the preconceptions of the journals’ editors to gain acceptance. The lesson is the oldest: people-even scientists!-easily believe what they want to.

(5) John Ioannidis and colleagues in their article ”Why Current Publication Practices May Distort Science.” The authors liken acceptance of papers in journals to winning bids in auctions: sometimes the winner pays too much and the results aren’t worth as much as everybody thinks. A review of the article here.

(7) UPDATE. Then there is arxiv.org, the repository of non-peer-reviewed ”preprints” (papers not yet printed in a journal). Arxiv is an acknowledgment by physicists, and lately mathematicians and even climatologists, that it’s better to take your findings directly to your audience and bypass the slow and error-prone refereeing process.

(8) It is easy to get a paper into print when the subject is ”hot”, or when you are friends with the editor or he owes you a favor, or your findings shame the editor’s enemies, or through a mistake, or by laziness of the referees, or in a journal with a reputation for sloppiness. In most fields, there are at least 100 monthly/quarterly journals. Thus it is exceedingly rare for a paper not to find a home, no matter how appalling or ridiculous its conclusions.

The listing of these facts is solely to prove that (A) and (B) are false, and that peer review is a crude sifter of truth.

Thus, when an activist or inactivist points to a peer-reviewed paper and says, ”See!”, he should not be surprised when his audience is unpersuaded. He should never argue that some finding must be true because it came from a peer-reviewed paper.

This web page has also tracked several peer-reviewed, published papers that are crap. Examples here, here, here, and here (more are coming).

————

1Incidentally, I have only had one methods paper rejected; all others I wrote were accepted to the first journal I sent them to. Nearly every collaborative paper I co-wrote has also been accepted eventually. I am an Associate editor at Monthly Weather Review, and have been a referee more times than I can remember, both for journals and grants.

I mention these things to show that I am familiar with the process and that I am not a disgruntled author seeking to impugn a system that has treated him unfairly. To the contrary, I have been lucky, and have had a better experience than most.

Läs även andra bloggares åsikter om <a href=”http://bloggar.se/om/milj%F6” rel=”tag”>miljö</a>, <a href=”http://bloggar.se/om/yttrandefrihet” rel=”tag”>yttrandefrihet</a>, <a href=”http://bloggar.se/om/fri-+och+r%E4ttigheter” rel=”tag”>fri- och rättigheter</a>varning-2

Annonser

Etiketter: ,

5 svar to “Peer Review – What it actually means 2”

  1. Roma Says:

    Just dropping by.Btw, you website have great content!

    ______________________________
    Professionally Written And Inspirational Wedding Speeches And Toasts…

  2. Robert Strong Says:

    News Letter from a Swedish Government Agency

    Not a critical view to the IPPC:s dishonest allegations on global warming as far as I can see. How is it possible? Sweden is still calculating on a CO2-taxation success at the Copenhagen meeting in December without asking a single serious question about critical scientific evidence of global warming issue? The Naturvårdsverket is a governmental organization that is paid for by the Swedish taxpayers. See their latest News Letter. I think it shows a cynical disrespect for ordinary people who expect to be wisely led by the democratically elected representatives.

    http://www.anp.se/newsletter.asp?sqid=227653&sid=42405B407948435F4B7740415C43&print=true

  3. Rise of sea levels is ‘the greatest lie ever told’ « UD/RK Samhälls Debatt Says:

    […] Peer Review – What it actually means 2 […]

  4. Fatal Errors in IPCC’S Global Climate Models « UD/RK Samhälls Debatt Says:

    […] Peer Review – What it actually means 2 […]

  5. The Origin and Life Cycle of Junk Science – OR Global Warming Hysteria « UD/RK Samhälls Debatt Says:

    […] Peer Review – What it actually means 2 […]

Kommentera

Fyll i dina uppgifter nedan eller klicka på en ikon för att logga in:

WordPress.com Logo

Du kommenterar med ditt WordPress.com-konto. Logga ut / Ändra )

Twitter-bild

Du kommenterar med ditt Twitter-konto. Logga ut / Ändra )

Facebook-foto

Du kommenterar med ditt Facebook-konto. Logga ut / Ändra )

Google+ photo

Du kommenterar med ditt Google+-konto. Logga ut / Ändra )

Ansluter till %s


%d bloggare gillar detta: