BBC website headline wrong shock horror

Bowel cancer screening 'does cut deaths', said the BBC News website today, in a report on a study using data from the NHS Bowel Cancer Screening Programme in England, published in the magnificently named journal Gut. Wow, I thought, that was quick, the programme has been going only since 2006 and didn't cover the whole country till 2010. Have they really found clear evidence of an effect on death rates already?

Well, no, they haven't. The story is a bit more complicated and subtle than the headline indicates.

There are several ways one can try to measure the success (or otherwise) of a screening programme for a life-threatening disease, but many of these can be misleading, and it's generally accepted that the only real way to measure the effectiveness of such a programme is to see what it does to the rate of mortality from the disease.

But it can take a long time to collect enough data to detect and measure changes in mortality rates. (Hence my surprise about the headline.) In deciding to introduce the screening programme, the NHS took account of the results of randomized trials of the screening which followed people up for long periods (in some cases well over 10 years) and did measure mortality. But these were conducted under specific conditions in a limited number of centres, and one cannot be sure that things will work the same way in a national programme.

On the basis of the trial results and various assumptions about how the national programme would work, it had been originally predicted that the NHS screening programme would reduce bowel cancer mortality in England by 16%. (In 2008, about 26 people in every 100,000 in England died of bowel cancer, so around 13,000 people in total - more data here.)

But it's important to understand that this new study did not measure mortality. It measured things like the numbers of people who took part in the screening programme as a percentage of those invited to take part, the results of the screening tests for those people, and the results of further investigations in people who were investigated further as a result of their screening results. (Overall, 2.5% of the men screened and 1.5% of the women had an abnormal screening test result.) These figures led the authors to conclude that the programme is still "on track to cut bowel cancer deaths by its target of 16%." That is, there was nothing to persuade them that the initial assumptions were wrong - but nothing at all to confirm the 16% figure directly.

All this is made perfectly clear in the study report itself. It's arguably a bit less clear in the press release, but that doesn't say that the screening "does cut deaths" and neither, I'd say, does the body of the BBC story.

So why does the headline say that the programme "does cut deaths", and in particular, why does it put that statement in quotation marks, as if someone other than the BBC had said it? The words in question are not in the original study report, not in the press release, and not in the quotations from various experts that are given in the body of the BBC report. So who actually said it? It matters, because it changes the importance of the story quite a lot.

Of course this might be just another example of statistical pedantry. But I think it's important - do you agree?

While I'm on, the story made a bit of a mess of some of the numbers it quoted, too. It says that the people screened were "aged 69 and over" - actually they were aged between 60 and 69 (though the scheme has recently been extended to older people). And it says that "Among the men with abnormal results, 43% turned out to have either cancer or pre-cancerous growths in their colon. The figure for women was 11.6%." No it wasn't. The figure for women was 29% - the 11.6% figure does appear in the press release, referring to something different. Maybe, by the time you read this, the BBC will have noticed these errors and fixed them.