Sceptical humility and peer review in science

This follows on from my recent post about Rebecca Watson’s condemnation of evolutionary psychology (see Sceptical arrogance and evolutionary psychology). Rebecca has now delivered this lecture several times New Zealand. None of them local so I couldn’t personally check if she had taken criticism on board. However, she was interviewed this morning on Kim Hill’s Saturday Morning (here’s the mp3 link).

She has not withdrawn her overall criticism of evolutionary psychology. She makes clear in her interview that this, as well as pop psychology in the media, are her targets (I’ll come back to that below). In fact, I think she makes her position even more untenable by providing a very naive description of peer review in science.

I wish sceptics when they defend science would describe it more realistically. It doesn’t help when they describe a utopian version which doesn’t exist in reality.

What is peer review?

This is an important social process in science where scientists’ ideas, conference presentations, and publications are reviewed by others in their field. Their peers. This helps reduce, maybe even eliminate, the influence of biases and pet, but unsubstantiated, theories held by the author. I have pointed out before that we are all prone to such cognitive biases – it’s part of being human. And having a Ph. D. doesn’t eliminate this human foible.

Scientists are human. And individually it’s hard to escape from our biases. The social nature of science helps to reduce their effect. Hence the importance of peer review.

But I must stress, such review doesn’t just operate at the time of a paper’s publication as Rebecca asserted in the interview. And it’s not just carried out under the watch of the scientific journal. Peer review occurs at all stages.

Peer review is occurring when hypotheses and ideas are being floated with colleagues. In more formal settings like departmental seminars, ideas are presented and exposed to criticism and improvement. Effectively conferences do the same thing to presentations. But often preparation of a conference presentation will have been reviewed by institutional colleagues – formally and informally. (Many groups will even have practice “dry runs” where the scientific content may be considered by colleagues as well as the details of the presentation, speech and visual aids).

Now, getting around to publications (the only area Rebecca included in her naive description). Good scientific institutes will have procedures which ensure formal internal review well before the paper is sent to the journal. And good journals will also have formal procedures to ensure quality and scientific standards in what they accept. Philosopher Masimmo Pigliucci provided a details of the procedure he uses as a journal editor in one of his Rationally Speaking podcasts (see 57: Peer Review). If you aren’t familiar with the process it’s worth listening to. Often such review will involve three anonymous referees, with the requirement that authors respond to questions and recommendations and final decisions on acceptance are made by an editor.

But wait, that’s not all.

The scientific peer review has barely started. Once published the research and conclusions are exposed to a far greater audience of peers. There’s plenty of opportunity for acceptance or rejection by peers. Often journals well accept “Letters to the Editor” types of response. Other scientists will condemn or support those conclusions when they write discussions in their own publications. There is scope for independent people to repeat the work, or usually something similar rather than exactly the same, and publish different conclusions.

Science is dynamic – our knowledge improves all the time. Publications are not sacred – they are easily and often superseded. (And there is scope for withdrawal of published papers when mistakes or scientific fraud are found).

It’s a mistake to think a published paper is the final authoritative stamp of approval on scientific ideas. It isn’t. And that’s the mistake Rebecca makes in her naive reference to the concept of peer review in science.

Rebecca presented an idealistic version of peer review where all mistakes, particularly scientific one, are detected during prepublication review of a paper. She says such mistakes should never make it into the published version. Yet, she says, this is happening in evolutionary psychology and she gives specific example where she critiques research findings and not just media coverage.

Well, guess what Rebecca. Such mistakes are probably made to some degree in all scientific fields. We are human after all. Mistakes do get into published papers (one of mine has my own name spelt wrong – five times). And all publishing scientists are well aware that some journals have much lower standards than others. We have probably all had a paper accepted without any feedback or criticisms from reviewers. Maybe even just on the decision of the editor. I certainly downgrade my impression of the journal when it happens to me.

Those shonky studies

Personally, it think peer review during publication may be a particular problem in the “soft sciences.” At least, I have been surprised to see some ideas presented in this area without supporting evidence, or obviously selected references. Perhaps these weaker standards are inevitable in some areas. Perhaps this allows more scope for intrusion of “political correctness” and popular ideological positions. Or perhaps authors feel less need to justify ideas if they are consistent with the prevailing ideologies in their field or institutes. Maybe the ideological issues in these areas are just too harsh to handle objectively. I imagine this might be true for feminism in the US and race relations in New Zealand.

I am sure Rebecca can find evolutionary psychology research journals where the quality of review is poor or ideologically compromised. But I am sure she could also find, if she looked, journals and publications where the standard of peer review is much higher. Perhaps her interest in feminist ideology and preoccupation with sex-related research has soured her overall view. I wouldn’t like to make that judgement. But soured it is.

Evolutionary psychology is being targeted

Some US bloggers have defended Rebecca on this issue by claiming her criticisms were only of pop psychology and media presentation. They refuse to acknowledge her inclusion of the whole field of evolutionary psychology in her attacks. Or else they excuse it. Maybe that is just the humane propensity to defensiveness coming out. Those sceptics may just be guilty of motivated reasoning (I referred to this in Sceptical arrogance and evolutionary psychology). For their sake I include this slide from Rebecca’s talk where she specifically describes her version of evolutionary psychology and critiques it.

I understand evolutionary psychology in broad terms as the application of an evolutionary perspective to human and animal psychology. This doesn’t need that researchers assume that human evolution stopped in the Pleistocene – or any of the other bullet points she has.

Rebecca has set up a straw man version of evolutionary psychology. Maybe that’s because of limitations in her reading or understanding. Maybe just because of her preoccupation with feminism and gender issues. But a straw man nevertheless.

Peer review for Rebecca

Rebecca Watson would have benefited immensely from some peer review herself before finalising her presentation. And all is not lost. Her presentation is getting peer review now. Yes, some of it will be rubbish which she should ignore. But there are some excellent comments being made she would be wise to take on board.

See also:
Science denialism at a skeptic conference
Science Denialism? The Role of Criticism
Oh gob, evo psych again?
Evolving skeptic psychology
Responsible Reading
Responsible Writing
FTB Blogger Stephanie Zvan Makes A Small Mistake
Let’s Confirm Negative Stereotypes About Women
αEP: Shut up and sing!
Do You Need To Be An Expert To Criticize Science?

Similar articles