The importance of what you don’t see

As of the 23rd May 2022 this website is archived and will receive no further updates. was produced by the Winton programme for the public understanding of risk based in the Statistical Laboratory in the University of Cambridge. The aim was to help improve the way that uncertainty and risk are discussed in society, and show how probability and statistics can be both useful and entertaining.

Many of the animations were produced using Flash and will no longer work.

Remember all those autism stories over the last few weeks? You don’t? There’s a reason for that.

The first story you did not hear about concerned a claimed association between autism and aerial crop-spraying of pyrethroids, arising from a press release from the American Association of Pediatrics of a non-peer-reviewed poster at a conference. The Science Media Centre asked for quotes from their cohort of scientists while the story was still embargoed, and the comments received were so uniformly negative that it received no coverage in the UK. Even in the US there were only limited reports.

The study did not deserve coverage at this stage. There was no accompanying paper and the poster featured the statistical conclusions and not the raw numbers of cases, which is always a suspicious sign. Furthermore, the analysis was apparently incorrect (see note below).

The second story associated autism with excess folic acid, and even though it was again a non-peer-reviewed conference paper, it was press-released by Johns Hopkins University, which as an extremely respectable institution aroused journalistic interest. Again it reported the relative risks, but did not report the actual number of cases involved. And again it got utterly hammered by the SMC’s panel, which kept it off the BBC, and the UK outlets that covered it included many highly-critical comments. The US coverage was fairly extensive but again fairly cautious.

I take a few lessons from this saga.

• We should be grateful for the general integrity of science correspondents both in the UK and the US, who spend as much of their time keeping stories out of the news as getting them in. And the Science Media Centre deserves a medal.

• The American Association of Pediatrics and Johns Hopkins University should be ashamed of themselves for press-releasing these stories based on a non-peer-reviewed conference poster and presentation. Autism is news, but it seems irresponsible to promote preliminary findings in such an important, but contested, area.

• Researchers, their employers and their professional organisations have extremely strong reasons for getting work publicly promoted. Standard ‘conflict of interest’ statements focus on narrow commercial interests and are inadequate to deal with this.

• It’s extremely difficult to think of what you are not seeing. But it's valuable to understand the process by which science stories appear in the media, and remember that most of them are there because someone wanted them to be there, and studies suggest that the majority of bad science in the media can be traced back to the press release.

Note on crop-spraying study. There were 24 zip-codes studied - 8 aerial sprayed with 19,000 children, and 16 not sprayed with 45,000 children. But the data appear to have been analysed as if 64,000 children have been individually ‘assigned’ to aerial spraying or not. This clustering by zip-code should be allowed for in the analysis, and so the reported measures of statistical significance appear incorrect, invalidating the conclusions.