Does street lighting really reduce fatal road crashes by 2/3 ?

Cochrane Reviews are usually taken as the gold standard in putting the evidence together to check whether a treatment works. But a new Cochrane Review that examines how much the ‘treatment’ of putting in street lights prevents injuries and saves lives seems to suffer from some major flaws which could mean the claimed benefits from street lighting are greatly exaggerated.

The Press release claimed that the Review showed that putting in street lights “reduced total crashes by between 32% and 55% and fatal injury crashes by 77%". This was picked up by the Daily Telegraph which reported that Turning off street lights could put motorists at risk, while the Daily Mail headlined with Switching off street lamps 'could triple road deaths'.

The actual review is much more cautious in its conclusions, but perhaps not cautious enough. It points out that the quality of the studies were poor and most were carried out decades ago when road conditions were different, with one even dating from 1948, 61 years ago. The evidence for the reduction in fatalities was 3 studies which looked at roads in which street lights had been installed, and compared the change in the number of fatal crashes during daylight with the change in fatal crashes during darkness: this comparison should mean that any trend over time is taken care of. Before the street lights were installed there had been 32 fatal crashes during daylight and 64 during darkness: after street lights there were 36 during daylight. If the street lights had no effect we would have expected there to be around 64 x 36/32 = 72 fatal crashes in darkness after the street lights were installed, whereas in fact there were only 25: 35% of what would have been expected had street lights been useless. So we might estimate a 65% reduction (the authors estimate a 67% reduction using a slightly different method – where the press release got their 77% reduction is a mystery).

This sounds a substantial reduction, but can we believe it? The authors acknowledge the poor quality of the studies, but appear to ignore two vital problems that make this estimate unreliable. The first is ‘publication bias’: if the street lights had had little or no effect, would the authors have gone to the effort of writing up their work and submitting it for publication? We can suspect that only strongly positive studies would have been published, especially decades ago.

The second point is more subtle and involves the concept known as ‘regression-to-the-mean’. Why were these particular roads chosen to have street lights installed? It is extremely likely that these stretches of roads had recently seen high rates of accidents. Accidents follow an erratic random pattern, with runs of highs and lows, and if roads are chosen because of a ‘high-blip’, it is likely that such a run of bad luck will come to an end and the accident rates will return to a more average figure. In other words we can expect the rates to improve by chance alone, whether or not street lights were installed. That’s why, if we really want to check if some new intervention works, we have to decide at random who gets it and who doesn’t.

This means that quoted benefits are likely to be considerable over-estimates, and to its great credit this was pointed out by the Daily Mail. it is a shame that this Review does not seem to have met the usual Cochrane standards, although it is not helped by an over-enthusiastic press release.