Convicted on Statistics?

by Vincent Scheurerⁱ

"… we do not convict people in these courts on statistics.
It would be a terrible day if that were so."
Mr Justice Harrison, R v. Sally Clark. November 1999ⁱⁱ

Introduction

On the 9th November 1999, Sally Clark, then a solicitor and 35-year-old mother of one, was convicted of the murder of her first two children. Christopher, her first child, had died three years earlier, aged just under three months. His death had originally been treated as arising by natural causes, probably “Sudden Infant Death Syndrome” (or “SIDS”). Her second child, Harry, died just over a year later at the age of two months. His death was treated as suspicious by the Home Office pathologist who carried out the post mortem. He then revisited Christopher’s death and determined that that too was suspicious. Sally Clark was arrested some weeks later and eventually tried at Chester Crown Court.

Since there were no witnesses to the actions Sally Clark was accused of, the evidence against her consisted primarily of evidence provided by a succession of highly qualified medical experts called by both the prosecution and the defence. The experts called by the prosecution looked at the medical evidence available, including the autopsy reports for both babies, and stated that these proved that both deaths were caused deliberately, either by shaking or smothering. Since Sally Clark was the only person with each child when he died, this evidence pointed to her. The experts called by the defence argued instead that the evidence was not conclusive and that the causes of the two deaths were simply unclear. At the end of the trial she was convicted of murdering both of her children and received two life sentences. She appealed to the Court of Appeal but her appeal was dismissed on the 2nd October 2000. It later transpired that essential evidence which could have provided an innocent explanation of Harry’s death had not been disclosed to the defence. Following the discovery of this evidence she appealed a second time and in January 2003 her appeal was allowed and she was set free. It is now widely recognised that she was a victim of a spectacular miscarriage of justice. However, it appears that the experience caused lasting damage, and she died on the 16th March 2007 due to acute alcohol intoxication.ⁱⁱⁱ

The trial and first appeal were also flawed in another, equally important respect. During the trial, statistical evidence and evidence relating to probabilities was given by one of the prosecution witnesses, Professor Sir Roy Meadow. This evidence was flawed, and although it is not possible to know whether this convicted Sally Clark, the judges at the second Court of Appeal stated that they would have reversed Sally Clark’s conviction on these grounds alone, even in the absence of withheld medical evidence iv. On closer examination, some of the mistakes made were simply astonishing; all the more so since they were made (and not corrected) in the full glare of a very public murder trial and subsequent appeal. These mistakes were shared by prosecution witnesses, the prosecution and defence counsel, the trial judge, the three judges at the first appeal and very possibly the trial jury. And yet they were mistakes of the most basic kind, which any person with a limited understanding of probability theory would almost certainly have avoided. This article will consider four of these mistakes which arose in connection with the evidence of Professor Meadow, one of the principal expert witnesses who appeared for the prosecution.

Professor Sir Roy Meadow and the CESDI report

At the time of the trial, Professor Meadow was a highly respected expert in field of child abuse, the author of a leading textbook “The ABC of Child Abuse”, and Emeritus Professor of Paediatrics and Child Health at St James’s University Hospital in Leeds. His evidence in the trial of Sally Clark covered much of the medical evidence relating to both deaths, which he concluded suggested that neither child had died a natural death^v.

However, one crucial section of his evidence was not of a medical nature at all. At the time of the trial, Professor Meadow was writing a preface to a report of a government funded multi-disciplinary research team, the “Confidential Enquiry into Sudden Death in Infancy” (or “CESDI”), entitled “Sudden Unexpected Deaths in Infancy”. This report covered an extensive study of infant deaths in the UK over a number of years, and had been undertaken in order to try to establish possible risk factors and associations for sudden and unexpected infant deaths. The report considered over 400 sudden infant death cases in the UK over a period of three years and, on the basis of these cases, identified a number of factors which might be associated with an increased risk of sudden unexpected infant death in a particular household. These factors included the presence of smokers in the relevant household, the age of the child’s mother, and whether the household included a wage-earner. The CESDI study, in its draft form at the time of Sally Clark’s trial, included the following table^vi:

Professor Meadow explained that the CESDI study showed that the risk of a child dying of SIDS in a household where none of the three risk factors listed above was present was 1 in 8,543 live births. It followed, therefore, that in order to quantify the risk of two children dying of SIDS in a household where none of these factors was present:

“you have to multiply 1 in 8,543 times 1 in 8,543 and I think it gives that in the penultimate paragraph, its (sic) points out that it’s approximately a chance of 1 in 73 million” ^viii

which would happen “by chance” once every 100 years. Put another way:

“… it’s the chance of backing that long odds outsider at the Grand National, you know; let’s say it’s a 80 to 1 chance, you back the winner last year, then the next year there’s another horse at 80 to 1 and it is still 80 to 1 and you back it again and it wins. Now here we’re in a situation that, you know, to get to these odds of 73 million you’ve got to back that 1 in 80 chance four years running, so yes, you might be very, very lucky because each time it’s just been a 1 in 80 chance and you know, you’ve happened to have won it, but the chance of it happening four years running we all know is extraordinarily unlikely. So it’s the same with these deaths. You have to say two unlikely events have happened and together it’s very, very, very unlikely.”

To the outside observer, the medical evidence in the trial of Sally Clark was extremely complicated. A number of different medical issues were debated by no fewer than nine specialists, who often reached different and contradictory conclusions. What is striking about the case is that, within this sea of complexity, the staggering figure of 1 in 73 million stands out like a beacon of simplicity. Unfortunately for Sally Clark, far from being a lighthouse to the truth, this figure managed the feat of being both irrelevant and wrong.

The mistakes

The first mistake that emerges directly from the paragraph quoted above is the assumption of independence. According to Professor Meadow, the risk of one SIDS death in a family without any of the three risk factors cited above was 1 in 8,543. So, he said, the probability of two SIDS death in a single family without any of these risk factors was simply 1 in 8,543 squared, or about 73 million; just as the probability of throwing a six is 1 in 6, and the probability of throwing a six followed immediately by another six is 1 in 36. He assumed that the death of Christopher Clark was independent of the death of Harry Clark. However, assuming that the figure of 8,543 was correct for one death, it clearly does not follow that 73 million is correct for two deaths. This would only be true if SIDS occurred randomly, without any chance of there being an unknown, hidden cause which might make a particular family more vulnerable to SIDS. Yet, and by its very nature, the cause (or causes) of SIDS is (or are) unknown. It was simply not possible for anyone to assume that SIDS is caused by random factors which are wholly independent of one another. Indeed, in a criminal trial, if the prosecution wishes to cite figures of this nature then it must prove – beyond all reasonable doubt – that the two events are entirely independent^ix. This was just not possible in the circumstances.

Unsurprisingly, this was a point made by the authors of the CESDI study themselves, who stated that the figures “did not take account of possible familial incidence of factors other than those included” in the table. The report went on to say:

“When a second SIDS death occurs in the same family, in addition to careful search for inherited disorder, there must always be a very thorough investigation of the circumstances- though it would be inappropriate to assume maltreatment was always the cause”.

It must surely be obvious to any person, whether a layman or a professional, that unexplained infant deaths may well have a genetic or hereditary cause which means that particular families are more likely to be afflicted by SIDS than the “average” family. The act of squaring the probabilities in order to reach the figure of one in 73 million was wholly illegitimate. Unfortunately, when this mistake was drawn to the attention of the judges at the first Court of Appeal hearing, they dismissed it as “[in]capable of affecting the safety of the convictions”^x.

An article published in the British Medical Journal some weeks after the trial cited other studies which suggested that SIDS deaths are not random events, and that “recurrence” of SIDS in the same family would be much more frequent than 1 in 73 million live births. Recurrence might even occur in England about once every 18 months, rather than once every 100 years^xi .

Prosecutor’s Fallacy

The second mistake arising from Professor Meadow’s evidence as to probabilities is known as the “Prosecutor’s Fallacy”. This consists of showing that the “innocent” explanation for certain facts is highly improbable – and then deducing that the “guilty” explanation is therefore the correct one. This type of mistake is particularly likely to occur in trials such as that of Sally Clark, where there are only two realistic options open to the jury – either the defendant committed murder, or her children died from unexplained, and comparatively rare, natural causes. For Sally Clark, once the jury accepted that there was only a 1 in 73 million chance that the deaths occurred due to SIDS, it was a short and easy step to conclude that this was the same probability as the probability of any other natural (but unexplained) cause of death; and that there was therefore a 1 in 73 million chance that she was not guilty of murder. It would be hard not to convict in those circumstances.

But, again, this reasoning is simply wrong. Even if we assume (for the purposes of this section, at least) that the probability of both of Sally Clark’s children dying due to natural causes was 1 in 73 million, this is simply of no assistance in trying to work out whether they actually died of natural causes or foul play. The fact that it is unlikely that a particular event will occur is not relevant when, after that event, one is trying to work out the cause. Once it is known that the two children are dead, the relevant question is not: “what is the probability that these deaths were natural?” but “is it more likely that these deaths were natural rather than deliberate?” In order to answer this question, the court would also have needed to assess the probability of the alternative explanation – of a mother murdering her first two children – and to compare that with the probability of two natural deaths. Common sense suggests that double murders by natural parents are also highly unusual; and indeed one of the witnesses^xii who provided evidence during the pre-trial “committal proceedings” stated that, according to his own research into “repeat” unexpected infant deaths (i.e. second deaths in the same family):

• one third were caused by rare but known natural causes (therefore not SIDS) missed by the experts performing the necropsy;
• one third were associated with child abuse; and
• one third were true “SIDS” deaths.

So fully two thirds of the cases of repeat deaths arose due to natural causes, and only one third due to foul play. Looking solely at the data for sudden unexpected deaths, the relevant probability was therefore a 2 in 3 chance of innocence, rather than a 1 in 73 million chance of innocence. The 1 in 73 million statistic was simply not relevant. Separately, Professor Philip Dawid, an expert witness called by Sally Clark’s team during the appeals process, pointed out that applying the same flawed method to the statistics for infant murder in England and Wales in 1996 could suggest that the probability of two babies in one family being murdered was 1 in 2,152,224,291 (that is one in 2.152 billion)^xiii; a sum even more outlandish than 1 in 73 million. The first Court of Appeal was alive to this potential argument, but preferred to ignore its implications, stating that “… the exercise [of expressing the probability of infant murder by a mother] is not an exercise the courts would perform”^xiv.

The first Court of Appeal accepted that this mistake had been made, and that the 1 in 73 million statistic, even if correct, should never have been introduced as it was simply not relevant to any of the issues which the jury had to decide. However, the appeal judges found that this was “merely a distraction” and was outweighed by the other evidence of guilt, which was “overwhelming”. As far as they were concerned
“If there had been no error in relation to statistics at the trial, we are satisfied that the jury would still have convicted on each count. In the context of the trial as a whole, the point on statistics was of minimal significance and there is no possibility of the jury having been misled so as to reach verdicts that they might not otherwise have reached. Had the trial been free from legal error, the only reasonable and proper verdict would have been one of guilty.” ^xv

This conclusion is clearly open to dispute. In a case with extremely complex, and heavily contested, medical evidence, it is not outlandish to wonder whether the jury was influenced by the statistic of 1 in 73 million, or the related concept of successfully backing an outsider to win the Grand National four years running. Far from being a distraction, it is entirely possible that this statistic was determinant.

The wrong expert

The two mistakes described above were sufficient on their own to justify a successful appeal in 2003. But at least two further errors were committed when the original trial heard the evidence of Professor Meadow.

Firstly, imagine that a statistician had been called to give evidence during the trial, and that her evidence included her opinion as to the presence, and relevance, of intra-retinal and petechial haemorrhages detected on the body of one of the children (one of the many medical questions hotly disputed in the case). Her evidence would have been stopped, the jury would have been told to disregard it, and she would probably find it hard to obtain further work as an expert witness – all because her evidence clearly moved from her field of competence to a completely different area of expertise. However, the reverse appears to have been allowed. The first Court of Appeal accepted that Professor Meadow “does not claim to be a statistician”^xvi and yet seemingly had no objection to him providing statistical evidence during the trial. The consequence was the introduction of a fallacious and irrelevant (yet highly damaging) statistic into a murder trial.

Ecological Fallacy

Lastly, let us look again at the CESDI study which was the ultimate source of the 1 in 73 million statistic. The purpose of this study was to identify possible causes of unexplained and sudden infant death. It did precisely that, and had much to say about the possibility of an increased risk of death in households with a smoker present, or no wage-earner, or where the mother was under a certain age. These findings were probably of substantial value in trying to identify groups of families at higher risk of sudden infant death. However, it is unlikely that the study was ever intended to provide evidence in a murder trial, and its use in the trial of Sally Clark was not appropriate. The trial of Sally Clark consisted of a detailed investigation into a specific allegation brought against a named individual, while the CESDI study was a wide-ranging study of 470,000 live births and could not, by definition, consider every aspect of every birth (or even every infant death) within that group. It was not permissible to make any inference about the specific facts relating to Sally Clark or her children on the basis of the findings of the CESDI study. Although the Clark family fell within a particular group identified within the CESDI study – no smoker, at least one wage earner, mother over 27 – this did not mean that the probability of an unexplained sudden infant death in their household was 1 in 8,543.

The assumption that data which is correct for a group is also correct for every individual within a group – a mistake known as the “Ecological Fallacy” – was clearly erroneous. If this mistake had been addressed at the outset then the 1 in 73 million statistic may never have been introduced at all.

Conclusion

Sally Clark was convicted of a double murder in the glare of national publicity. Her first appeal was turned down under international press scrutiny. The professionals involved in her trial and first appeal, including the expert witnesses, lawyers and judges, were the best available under English law. And yet it appears that nobody noticed the significance of the errors I have described above. It is possible, as the first Court of Appeal stated, that none of these errors were relevant and that the jury was swayed solely by the medical evidence presented by the prosecution. But that evidence was extremely complex, and it is not difficult to suppose that the jury found the 1 in 73 million statistic helpful in deciding to convict Sally Clark. Equally, the jury cannot really be blamed for this. But the same cannot be said of the criminal justice system which allowed this statistic to poison a trial and then failed to correct the mistake even after it had been identified.

But does the case have any wider implications? Has anyone else been convicted by bad statistics, or was Sally Clark just desperately unlucky? In April 2002, Angela Cannings was convicted of murdering two of her children in a case which bore a strong resemblance to the case against Sally Clark, not least in the presence yet again of Professor Meadow as an expert witness for the prosecution ^xvii. Angela Cannings was released by the Court of Appeal in December 2003, after the second Court of Appeal freed Sally Clark; indeed, once Sally Clark had been freed, the case against Angel Cannings (and other cases against other women whose children had died of unknown causes, including Trupti Patel and Donna Anthony) simply became untenable.

But the wider question of whether mistakes of the kind made in the Sally Clark trial have been made in criminal cases of a very different type was answered on the 15th November 2007, when the Court of Appeal published its judgment in the case of R v Barry George.

Barry George

On the 26th April 1999, some three months before Sally Clark was charged with the double murder of her two children, Jill Dando was shot and killed outside her home in London. Ms Dando was a well known TV personality and her murder, committed in broad daylight, was immediately headline news. More than a year later, Barry George was arrested and later charged with her murder. Forensic scientists had found a single particle of firearm discharge residue inside the pocket of his coat, and this proved to be a core part of the prosecution’s case. At the trial, experts for the prosecution and the defence argued long and hard about whether there was an innocent explanation for the presence of this residue inside the coat, by “innocent contamination” (instead of contamination because the wearer of the coat had fired a gun). The prosecution expert witnesses addressed the probability of innocent contamination in detail and concluded that it was “most unlikely” (which may well have been true). On this basis, the prosecution asked the jury to ignore the possibility of innocent contamination, leaving the only alternative explanation that Barry George was guilty of murder.

Mr George was convicted and given a life sentence. Unfortunately, it later transpired that at least one prosecution expert witness considered that the probability of the residue being deposited through non-innocent means was equally unlikely.^xviii This, however, had never been made clear to the jury during the trial; and indeed the logical inference to be drawn from the prosecution case was that the non-innocent explanation was much more likely than the innocent explanation. When this came to light, the Court of Appeal had no alternative but to quash Barry George’s conviction for murder. A second trial was held during the summer of 2008, and more than seven years after he was originally convicted of murdering Jill Dando, Barry George was finally acquitted.

This was another example of the Prosecutor’s Fallacy in action. (i) The prosecution sought to persuade the jury that the presence of firearm discharge residue in Barry George’s coat pocket showed that he had fired a gun. (ii) In order to prove this, the prosecution showed that, before the event, it was highly unlikely that this residue would find its way into a person’s pocket by innocent contamination. (iii) At the trial after the event, the court was therefore invited to discount that innocent explanation, in favour of a guilty explanation. (iv) Nobody told the jury that the guilty explanation was just as likely as the innocent explanation; and it appears that nobody asked that question.

Once again, this happened in a high profile murder trial, under even more scrutiny than the trial of Sally Clark. Who is to say how many other miscarriages of justice have taken place, based on the Prosecutor’s Fallacy, which are yet to be corrected?

Some useful links

1. Judgments
   • First Court of Appeal judgment (Sally Clark)
   • Second Court of Appeal judgment (Sally Clark)
   • Second Court of Appeal judgment (Barry George)
   • Court of Appeal judgment (Angela Cannings)
   • Court of Appeal judgment (Donna Anthony)
2. Protagonists and Commentators
   • Information about the trial of Sally Clark and background data on cot deaths and related links can be found on the Sally Clark website.
   • The expert report in the Sally Clark case of Professor Philip Dawid of University College London can be found in this document.
   • The Royal Statistical Society issued a press release highlighting its concern about the Sally Clark trial in October 2001.
   • Stephen Watkins’ article in the British Medical Journal entitled Conviction by mathematical error can be found on the BMJ’s website (registration is required).
   • Professor Meadows’ telling reply in the British Medical Journal to Mr Watkins’ article, entitled A case of murder and the BMJ, can be found on the BMJ’s website (registration is required):
   • All of the major UK broadsheets, together with the BBC News website, have extensive online archives covering the trials of Sally Clark and Barry George.
3. Prosecutor’s Fallacy and other Fallacies

   Wikipedia is always a useful starting point for a summary of mathematical concepts.

   The source of the term “Prosecutor’s Fallacy” is an article by William Thompson and Edward Schumann (not available for free download) at http://www.jstor.org/pss/1393631.

   An interesting take on the Prosecutor’s Fallacy in a No Limit Texas Hold’Em hand can be found at http://pokersleuth.com/fallacy-prosecutor.shtml

4. Expert Evidence

   At the time of writing, the Law Commission is consulting on possible reforms to the rules governing expert evidence in criminal cases in England and Wales. Part 1 of the Law Commission’s Consultation Paper No. 190 sets out a convenient summary of the current rules on expert evidence and can be found at http://www.lawcom.gov.uk/docs/cp190.pdf. (It also cites the Sally Clark and Angela Cannings cases as an example of an “ongoing problem” with expert evidence, which needs an “urgent solution”).

5. DNA evidence

   The increasing reliance on DNA evidence in criminal trials means that highly complex issues relating to statistics and probabilities will continue to trouble lawyers and juries for the foreseeable future. An interesting analysis of some of the issues involved in considering the rarity of DNA profiles, by Bruce Weir, can be found at: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2585748

End notes

SCCA1 is the first Court of Appeal judgment in the case of R v Sally Clark.
SCCA1 is the second Court of Appeal judgment in the case of R v Sally Clark.
BGCA2 is the second Court of Appeal judgment in the case of R v Barry George.

1. With thanks to Ted Davison and Cécile Bouchet for their comments on the first draft.
2. SCCA1 at paragraph 128
3. Times Online article.
4. SCCA2 at paragraph 180
5. SCCA2 at paragraphs 46, 58 and 90
6. SCCA1 at paragraph 121. The central column is empty in the Court of Appeal judgment as well.
7. SCCA2 at paragraph 96
8. SCCA2 at paragraph 99
9. R v Turner [1975] QB 834 at page 840
10. SCCA1 at paragraph 142
11. “Conviction by mathematical error”, Stephen Watkins, BMJ Volume 320 January 2000, to which Prof. Meadow responded in volume 324, but without defending the statistics he used.
12. Professor Emery, see SCCA1 at paragraph 116
13. Paragraph 16 of Professor Dawid’s expert report
14. SCCA1 at paragraph 160
15. SCCA1 at paragraph 256
16. SCCA1 at paragraph 112
17. But without the 73m-1 statistic in this instance
18. BGCA2 at paragraphs 18 and 38

About the Author

Vincent Scheurer

Vincent Scheurer is a London-based lawyer with a long-standing interest in probability theory and statistics. He is currently writing a book on gambling and the law.