Statistician's ESP prediction comes true!

In our article on ESP and the significance of significance, I made a prediction on the controversy surrounding psychologist Daryl Bem's paper "Feeling the Future" - the one that claimed to find evidence for some forms of ESP (extra-sensory perception). "This one will run and run", I boldly prophesied. And so it came to pass.

The latest round in the controversy comes from psychologists Jeff Rouder (University of Missouri) and Richard Morey (University of Groningen). Their paper A Bayes factor meta-analysis of Bem's ESP claim has just been published online by the Psychonomic Bulletin & Review (behind a paywall, but there's a version on Rouder's university website here). My prediction was hardly a difficult one to make - versions of this paper have been around on the authors' websites for months.

This isn't the only development. There's been a fuss because the Journal of Personality and Social Psychology, that originally published Bem's study, has apparently refused to publish a paper by yet more psychologists, Christopher French, Stuart Ritchie and Richard Wiseman. The paper describes their attempts to replicate one of Bem's experiments - the replications did not succeed. Here is New Scientist magazine's report.

Rouder and Morey manage to criticize both Bem and also Bem's critic Eric-Jan Wagenmakers. They calculate Bayes factors for some of Bem's studies in a different way from either Wagenmakers and his colleagues or Bem, Utts and Johnson. (For details on all that, see my original article.) Rouder and Morey do calculate Bayes factors that provide a certain amount of evidence in favour on ESP, but conclude that there is not enough evidence to outweigh the prior beliefs of an "appropriately skeptical reader". They "remain unconvinced of the viability of ESP".

If you've been keeping up with all this, maybe you're feeling a little disturbed that three different groups have all used what looks like the same approach, Bayes factors, and have come up with different numbers and, in some cases, very different conclusions. Don't the data provide the answer, you might be wondering.

No, the data don't provide the answer. In situations of uncertainty, we need statistical methods to help us be rational and consistent. The statistics can tell us a great deal whether our beliefs fit together rationally, and about how they should change in the light of new evidence, but they cannot and do not tell us what to believe or what to decide. The three groups come up with different Bayes factors because those Bayes factors represent different prior beliefs, even though they all use the same data.

Finally, if you still haven't had enough of this, you might be interested in the May 2011 issue of the journal Perspectives on Psychological Science. This has a special section on Bayesian data analysis in psychology, including several papers that refer to the Bem study and one that consists solely of a rather involved Dutch joke about prior distributions (which is also available, outside the paywall, here).