Tag Archives: peer review

Sceptical humility and peer review in science

This follows on from my recent post about Rebecca Watson’s condemnation of evolutionary psychology (see Sceptical arrogance and evolutionary psychology). Rebecca has now delivered this lecture several times New Zealand. None of them local so I couldn’t personally check if she had taken criticism on board. However, she was interviewed this morning on Kim Hill’s Saturday Morning (here’s the mp3 link).

She has not withdrawn her overall criticism of evolutionary psychology. She makes clear in her interview that this, as well as pop psychology in the media, are her targets (I’ll come back to that below). In fact, I think she makes her position even more untenable by providing a very naive description of peer review in science.

I wish sceptics when they defend science would describe it more realistically. It doesn’t help when they describe a utopian version which doesn’t exist in reality.

What is peer review?

This is an important social process in science where scientists’ ideas, conference presentations, and publications are reviewed by others in their field. Their peers. This helps reduce, maybe even eliminate, the influence of biases and pet, but unsubstantiated, theories held by the author. I have pointed out before that we are all prone to such cognitive biases – it’s part of being human. And having a Ph. D. doesn’t eliminate this human foible.

Scientists are human. And individually it’s hard to escape from our biases. The social nature of science helps to reduce their effect. Hence the importance of peer review.

But I must stress, such review doesn’t just operate at the time of a paper’s publication as Rebecca asserted in the interview. And it’s not just carried out under the watch of the scientific journal. Peer review occurs at all stages.

Peer review is occurring when hypotheses and ideas are being floated with colleagues. In more formal settings like departmental seminars, ideas are presented and exposed to criticism and improvement. Effectively conferences do the same thing to presentations. But often preparation of a conference presentation will have been reviewed by institutional colleagues – formally and informally. (Many groups will even have practice “dry runs” where the scientific content may be considered by colleagues as well as the details of the presentation, speech and visual aids).

Now, getting around to publications (the only area Rebecca included in her naive description). Good scientific institutes will have procedures which ensure formal internal review well before the paper is sent to the journal. And good journals will also have formal procedures to ensure quality and scientific standards in what they accept. Philosopher Masimmo Pigliucci provided a details of the procedure he uses as a journal editor in one of his Rationally Speaking podcasts (see 57: Peer Review). If you aren’t familiar with the process it’s worth listening to. Often such review will involve three anonymous referees, with the requirement that authors respond to questions and recommendations and final decisions on acceptance are made by an editor.

But wait, that’s not all.

The scientific peer review has barely started. Once published the research and conclusions are exposed to a far greater audience of peers. There’s plenty of opportunity for acceptance or rejection by peers. Often journals well accept “Letters to the Editor” types of response. Other scientists will condemn or support those conclusions when they write discussions in their own publications. There is scope for independent people to repeat the work, or usually something similar rather than exactly the same, and publish different conclusions.

Science is dynamic – our knowledge improves all the time. Publications are not sacred – they are easily and often superseded. (And there is scope for withdrawal of published papers when mistakes or scientific fraud are found).

It’s a mistake to think a published paper is the final authoritative stamp of approval on scientific ideas. It isn’t. And that’s the mistake Rebecca makes in her naive reference to the concept of peer review in science.

Rebecca presented an idealistic version of peer review where all mistakes, particularly scientific one, are detected during prepublication review of a paper. She says such mistakes should never make it into the published version. Yet, she says, this is happening in evolutionary psychology and she gives specific example where she critiques research findings and not just media coverage.

Well, guess what Rebecca. Such mistakes are probably made to some degree in all scientific fields. We are human after all. Mistakes do get into published papers (one of mine has my own name spelt wrong – five times). And all publishing scientists are well aware that some journals have much lower standards than others. We have probably all had a paper accepted without any feedback or criticisms from reviewers. Maybe even just on the decision of the editor. I certainly downgrade my impression of the journal when it happens to me.

Those shonky studies

Personally, it think peer review during publication may be a particular problem in the “soft sciences.” At least, I have been surprised to see some ideas presented in this area without supporting evidence, or obviously selected references. Perhaps these weaker standards are inevitable in some areas. Perhaps this allows more scope for intrusion of “political correctness” and popular ideological positions. Or perhaps authors feel less need to justify ideas if they are consistent with the prevailing ideologies in their field or institutes. Maybe the ideological issues in these areas are just too harsh to handle objectively. I imagine this might be true for feminism in the US and race relations in New Zealand.

I am sure Rebecca can find evolutionary psychology research journals where the quality of review is poor or ideologically compromised. But I am sure she could also find, if she looked, journals and publications where the standard of peer review is much higher. Perhaps her interest in feminist ideology and preoccupation with sex-related research has soured her overall view. I wouldn’t like to make that judgement. But soured it is.

Evolutionary psychology is being targeted

Some US bloggers have defended Rebecca on this issue by claiming her criticisms were only of pop psychology and media presentation. They refuse to acknowledge her inclusion of the whole field of evolutionary psychology in her attacks. Or else they excuse it. Maybe that is just the humane propensity to defensiveness coming out. Those sceptics may just be guilty of motivated reasoning (I referred to this in Sceptical arrogance and evolutionary psychology). For their sake I include this slide from Rebecca’s talk where she specifically describes her version of evolutionary psychology and critiques it.

Evol-Psych

I understand evolutionary psychology in broad terms as the application of an evolutionary perspective to human and animal psychology. This doesn’t need that researchers assume that human evolution stopped in the Pleistocene – or any of the other bullet points she has.

Rebecca has set up a straw man version of evolutionary psychology. Maybe that’s because of limitations in her reading or understanding. Maybe just because of her preoccupation with feminism and gender issues. But a straw man nevertheless.

Peer review for Rebecca

Rebecca Watson would have benefited immensely from some peer review herself before finalising her presentation. And all is not lost. Her presentation is getting peer review now. Yes, some of it will be rubbish which she should ignore. But there are some excellent comments being made she would be wise to take on board.

See also:
Science denialism at a skeptic conference
Science Denialism? The Role of Criticism
Oh gob, evo psych again?
Evolving skeptic psychology
Responsible Reading
Responsible Writing
FTB Blogger Stephanie Zvan Makes A Small Mistake
Let’s Confirm Negative Stereotypes About Women
αEP: Shut up and sing!
Do You Need To Be An Expert To Criticize Science?

Similar articles

ID research and publications

Here is another post to mark Darwin Day.

The pro-intelligent design (ID) internet echo chamber has been making a big thing of late about “peer-reviewed papers supporting intelligent design.” Their “Center for Science and Culture” has even published an updated list. (PZ Myers has provided a more accessible version of the list at More bad science in the literature).

This of course does raise some questions about what they mean by “peer review” and the real nature of some of the journals these papers are in (have a look at their in-house journal Bio-Complexity). But leaving those issues aside for now I just don’t think any of these papers are reporting “ID research.”

The nature of “ID research”

To me research supporting intelligent design should postulate some structured hypotheses for ID and seek to test them or validate them against reality. But none of the articles do that. Most, especially ones that are published in credible journals, deal with aspects of evolutionary science.

Sure they may postulate a problem, an example or issue where they feel current science does not have an answer. That’s what I expect in a scientific paper. Identification of problems and reporting work on them.

Like all areas of science, evolutionary science has its so far unanswered questions, its problems and anomalies. perfectly natural and perfectly acceptable to identify and investigate them. But calling such work “supporting intelligent design” is just dishonest. No specific ID hypotheses have been advanced, let alone tested.

This always seems to be the case for any list of “peer-reviewed scientific papers supporting ID.”

“Theistic science” – or argument by default

Nor, by the way, do these papers display any example of the alternative to “materialist” science. Their declared aim of replacing modern science with a “theistic science.” (See Wedge Strategy and Theistic science? No such thing). If they were doing any work like this why isn’t that demonstrated by the publications? I would love to see examples of such research and identify the different methods characteristic of such science.

To list these papers as supporting ID  is simply assuming that any criticism, any problem, any gap in evolutionary science is, by default, evidence for ID.

It’s not.

Relying on cranks

David L Abel

Another issue with this publication list which does supply some mirth is the frequent occurrence of publications by David L. Abel (17% of total list). He has raised some attention because he published a paper in the journal Life which had recently received attention for its publication of the whaky paper Theory of the Origin, Evolution, and Nature of Life,” by Erik D. Andrulis. (See The comparison to jabberwocky is inevitable for PZ Myers’ in depth discussion of that paper). Abel’s article is titled “Is Life Unique?” – Myers describes this as “Intelligent Design creationism crap,” and “drivel” (see More bad science in the literature). But Myers was impressed with Abel’s address and affiliation:

Department of ProtoBioCybernetics and ProtoBioSemiotics, Origin of Life Science Foundation, Inc., 113-120 Hedgewood Drive, Greenbelt, MD 20770

Turns out this is a residential house, probably Abel is the only “employee,” but it does have na official name plate besides his front door! As PZ says:

“That’s every intelligent design creationism institute of scientific thinking: a cheap sign tacked up on a garage, with some guy with delusions of competence twiddling his thumbs inside.” (see Zooming in on the Origin of Life Science Foundation)

Abel himself describes his institute as a “science and education foundation with corporate headquarters near NASA’s Goddard Space Flight Center just off the Washington, D. C. Beltway in Greenbelt, MD.”  If you are not careful in your reading you might assume he was actually based at a NASA site!

And here is the information on Abel held in his profile at the ID journal
Bio-Complexity.

David L. Abel

Affiliation The Gene Emergence Project; The Origin-of-Life Science Foundation
Bio statement Director, The Gene Emergence ProjectDepartment of ProtoBioCybernetics & ProtoBioSemioticsThe Origin of Life Science Foundation, Inc.

These lists of “peer reviewed papers supporting ID’ are getting rather desperate.

Similar articles

Personal attacks on climate scientists

The American Association for the Advancement of Science (AAAS) released a statement this week expressing concern for the current personal attacks being made on climate scientists by politicians and others. The text of the statement follows:


Statement of the Board of Directors of the American Association for the Advancement of Science
Regarding Personal Attacks on Climate Scientists
Approved by the AAAS Board of Directors
28 June 2011

We are deeply concerned by the extent and nature of personal attacks on climate scientists. Reports of harassment, death threats, and legal challenges have created a hostile environment that inhibits the free exchange of scientific findings and ideas and makes it difficult for factual information and scientific analyses to reach policymakers and the public. This both impedes the progress of science and interferes with the application of science to the solution of global problems. AAAS vigorously opposes attacks on researchers that question their personal and professional integrity or threaten their safety based on displeasure with their scientific conclusions. The progress of science and protection of its integrity depend on both full transparency about the details of scientific methodology and the freedom to follow the pursuit of knowledge. The sharing of research data is vastly different from unreasonable, excessive Freedom of Information Act requests for personal information and voluminous data that are then used to harass and intimidate scientists. The latter serve only as a distraction and make no constructive contribution to the public discourse.

Scientists and policymakers may disagree over the scientific conclusions on climate change and other policy-relevant topics. But the scientific community has proven and well-established methods for resolving disagreements about research results. Science advances through a self-correcting system in which research results are shared and critically evaluated by peers and experiments are repeated when necessary. Disagreements about the interpretation of data, the methodology, and findings are part of daily scientific discourse. Scientists should not be subjected to fraud investigations or harassment simply for providing scientific results that are controversial. Most scientific disagreements are unrelated to any kind of fraud and are considered a legitimate and normal part of the scientific process. The scientific community takes seriously its responsibility for policing research misconduct, and extensive procedures exist to protect the rigor of the scientific method and to ensure the credibility of the research enterprise

While we fully understand that policymakers must integrate the best available scientific data with other factors when developing policies, we think it would be unfortunate if policymakers became the arbiters of scientific information and circumvented the peer-review process. Moreover, we are concerned that establishing a practice of aggressive inquiry into the professional histories of scientists whose findings may bear on policy in ways that some find unpalatable could well have a chilling effect on the willingness of scientists to conduct research that intersects with policy-relevant scientific questions.

Real science – warts and all

After the PR hype NASA seemed to purposely promote around “arsenic bacteria” research published by Science (see NASA and old lace) there has been quite a critical reaction. Critical of the way the story was hyped by NASA, but also critical of the work itself.

The whole story does raise issues of how science is done, how it is published and reviewed, and how it is reported in the media.  It also raises issues about the sometimes negative role institutions like NASA can play in all this.

There is a useful discussion of this on the latest Guardian Science Weekly podcast (see Science Weekly podcast: Global criticism of the arsenic bacteria study; plus, we expose some dating myths).

Download mp3

A panel of “those in the know,” including astrobiologist Dr Zita Martins from Imperial College London and science writer David Dobbs who has been blogging and tweeting about this specific research, discuss the issue. David writes for the Atlantic Monthly, New York Times Magazine, Slate, National Geographic, Audubon, and Scientific American Mind, where he is a contributing editor. There is also a clip from Carl Zimmer speaking on NHPR (New Hampshire Public Radio).

The discussion gives a good idea of how science is actually done – warts and all! It looks behind the sometimes ideally presented public image and considers the problem of scientists own emotional agendas, the reality of peer review and new issues arising from the way science is conducted in the internet age.

The panelists see any problems with the “arsenic bacteria” research being resolved over time by the normal process of science and stress that the issues discussed are more general.

As an extra, and for light relief, the podcast also contains  comments from Dr Petra Boynton from UCL exposing four key myths about dating.

The Guardian Science Weekly recently won an award for the best science podcast. It is well presented and informative. Worth subscribing to and following.

Similar articles

Enhanced by Zemanta

Putting the IPCC in its place?

My PhotoThe blog The Climate Scum is worth keeping an eye on. It’s satirical, of course, but it’s content is not too far from what we often find on the internet. By the way it’s written by Baron von Monckhofen (right).

Here’s an extract from a recent post Reforming the IPCC: how to do it properly!

“The following measures are intended to turn the IPCC and its future assessment reports into vehicles for Truth and Reason instead of vehicles for Eco-Fascist Fraud and Deception, as they have been so far.

  1. No communists like Hansen and Mann should be allowed to participate. Only politically independent and objective people should be allowed. Thus, alls participants must have read and memorized “Atlas Shrugged”.
  2. No people who receive grants for doing climate science should be allowed to participate. Such people will just make up scary things so they can get even more grants. Only economically independent people should be allowed.
  3. Likewise, no people who publish climate science articles in peer-reviewed journals should be allowed. They just want to cite their own papers and those of their tribe.
  4. No Chinese or Indians, who just want to weaken the competiveness of the West. Tricky bastards!
  5. No previous IPCC participant can participate in the new IPCC (in particular not Pachauri)! . As everybody who has any experience with management knows, if you want to change an organization the first thing you must do is to get rid of all members/employees.
  6. All previous IPCC participants must release all the email correspondence they have ever had. Releasing email correspondence is vital for the auditing of science and to guarantee repeatability and transparency.
  7. All IPCC prisoners must be released and all weapons of mass destructions must be disarmed.
  8. Any IPCC participant that claims that CO2 can affect the climate must, in order to be credible, abstain from travelling in airplanes and in cars, living indoors, eating warmed food and breathing.
  9. No use of models. Good science is based on empirical observations, and not models. In particular, any “predictions” and “projections” about the future must be entirely based on observations, and not models. If Galileo and Newton and Maxwell and Einstein had bee diddling with models, science would never have progressed.
  10. No use of temperature data. Temperature data, whether from thermometers on the ground or those mounted on satellites, are notoriously unreliable and affected by the urban heat island effect.
  11. Likewise, sea level data, carbon dioxide data, precipitation data, arctic ice volume data and climate proxies must be avoided, as they are inherently unreliable and unscientific.
  12. Climate data from other planets must be included, so we can compare the warming on Earth, the Moon, Mars, Jupiter, Haley’s comet and the iron-core Sun. No theory that cannot explain all these warming incidents should be taken seriously.
  13. Anecdotal evidence, such as medieval Chinese fleets navigating around the North Pole, should not be dismissed unless proven wrong beyond the shadow of a doubt. To rely more on instruments than on human observers and chroniclers is elitistic and in its essence anti-human.
  14. No references should be allowed to any shady grey literature, like WWF reports. Only shiningly white NGOs working for the benefit of mankind, like the Heartland Institute, should be referenced. White humans are more important than grey frogs!
  15. No references should be allowed to journals like Nature and Science, which have been participating in the suppression of AGW-skeptical papers. Only truly openminded and unbiased journals like Energy & Environment should be referenced.
  16. For each unbalanced alarmist reference, there must be at least one skeptical reference in order to assure fairness and balance.
  17. Uncertainty should be specified according to the scale “Uncertain”, “Highly uncertain”, “Extremely uncertain” and “Completely wrong”.
  18. The best science nowadays is done on blogs, were new ideas easily can be proposed and peer review is instant. Hence, the focus of the assessment reports should be moved from reviewing what is published in the ivory-tower journals to what is published on the blog science blogs. The blogs belonging to journals like Science and Nature do not count – they are just ivory tower blogs masquerading as blog science blogs.
  19. The assessment reports should not exceed 20 pages, and all information should be presented as comic strips. In that way, even illiterate people with a limited attention span will be able to comprehend it. (Like Al Gore, he he!)
  20. In order to ensure its independence, the IPCC should not receive any funding from governments. Instead, it has find its own financing, for instance by selling advertisements in the assessment reports. The taxpayer money that is saved can be used for more important things, like eradicating malaria and giving tax cuts for productive citizens.”
Enhanced by Zemanta

A desperate plea to be noticed?

Quite a few local bloggers* have commented on the legal action some New Zealand climate deniers are taking to get NIWA to change its national temperature record. This is only the latest step in a nasty little campaign by these people to deny the reality of climate change. Nasty because it distorts the data and facts and makes outrageous attacks on the integrity and honesty of New Zealand scientists. The latest step – but I do wonder if it is the last step – seeing it is likely to backfire.Initially this campaign attempted to take advantage of the “climategate” email hysteria to whip up local anti-science feelings. Of late, as this hysteria has dispersed the local deniers have deteriorated to a small but vocal clique making carping and dishonest attacks on NIWA. I guess they see this legal action as a way of somehow revitalising their campaign.

Continue reading

Creationism, climate change and scientific denialism

Strange how one keeps coming across familiar people. Last week I was shocked to read Andrew Bolt’s article on the Global Atheist Convention (see Speakers true love of hatred). He wasn’t there and those of us that were recognised his article as full of distortions, misinformation and lies. Then I realised he is one of Australia’s most active climate change deniers (no, in his case not a sceptic – a denier). And now I find him quoted by the creationist/intelligent design (ID) Discovery Institute in a nasty little article attacking science (see Climategate: a Word of Advice to the Scientists).

Continue reading

Freedom of information and responsibility

Prepare for another round of “climategate,” misinformation and distortion of climate change news. This time conservative news media, denier organisations and bloggers will be concentrating on the current inquiries taking place in the UK. These will be considering issues related to the illegal hacking and release of emails from the Climate Research Unit and the University of East Anglia. One of these is the Science and Technology Committee of the UK Parliament inquiry which hears oral submissions this week.

Already we have seen some selective and biased reporting of written submissions and I am sure this will continue. However, there are important issues at stake to do with freedom of information (FOI), harassment of scientists and the responsibility of those making freedom of information requests or using publicly available data. I hope the inquiries will deal with the underlying principles as well as making determinations on the specific cases considered.

Clearly many of the FOI requests made to the CRU were malicious. The UEA submissions says:

“In July 2009 UEA received an unprecedented, and frankly administratively overwhelming, deluge of FOIA requests related to CRU. These amounted to 61 requests out of a 2009 total of 107 related to CRU, compared to annual totals of 2 in 2008 and 4 in 2007 (University totals for those years were 204, 72 and 44 respectively).”

I wonder if the requesters were building up to something?

(Graphics thanks to Going on a Bear Hunt).

Continue reading

Beware the retired scientist?

The Lippard Blog has an interesting analysis of Who are the climate change skeptics? In this he identifies links of many of the sceptics with several right wing think tanks like The Heartland Institute and George C. Marshal Institute. One could do a similar analysis of our local climate sceptics and deniers. Some of them seem to be linked with the right wing NZ Centre for Politcal Research, the ACT Party and Conservative Christian organisations and blogs. Have a look at the discussion New Zealand’s “CLIMATEGATE”! on the Centre for Political Research forum. Obviously conspiracy theorists tend to congregate in these areas.

Continue reading

Overcoming dogmatism in science

One more post from 18 months ago:

Karl PopperAn important element of the scientific method is that hypotheses must be testable, potentially falsifiable, to be scientific. That we build theories by testing hypotheses and rejecting them if proved wrong by experiential evidence. And not just hypotheses. Prevailing theories are also constantly open to potential falsification, testing against new evidence and changing to incorporate new findings.

The concept of falsification in science was popularised by Karl Popper.

Continue reading