Court of Appeal bans Bayesian probability (and Sherlock Holmes)

david's picture
Submitted by david on Mon, 02/25/2013 - 09:26

..when you have eliminated the impossible, whatever remains, however improbable, must be the truth
(Sherlock Holmes in The Sign of the Four, ch. 6, 1890)

In a recent judgement the English Court of Appeal has not only rejected the Sherlock Holmes doctrine shown above, but also denied that probability can be used as an expression of uncertainty for events that have either happened or not.

The case was a civil dispute about the cause of a fire, and concerned an appeal against a decision in the High Court by Judge Edwards-Stuart. Edwards-Stuart had essentially concluded that the fire had been started by a discarded cigarette, even though this seemed an unlikely event in itself, because the other two explanations were even more implausible. The Court of Appeal rejected this approach although still supported the overall judgement and disallowed the appeal - commentaries on this case have appeared here and here.

But it's the quotations from the judgement that are so interesting:

Sometimes the "balance of probability" standard is expressed mathematically as "50 + % probability", but this can carry with it a danger of pseudo-mathematics, as the argument in this case demonstrated. When judging whether a case for believing that an event was caused in a particular way is stronger that the case for not so believing, the process is not scientific (although it may obviously include evaluation of scientific evidence) and to express the probability of some event having happened in percentage terms is illusory.

The idea that you can assign probabilities to events that have already occurred, but where we are ignorant of the result, forms the basis for the Bayesian view of probability. Put very broadly, the 'classical' view of probability is in terms of genuine unpredictability about future events, popularly known as 'chance' or 'aleatory uncertainty'. The Bayesian interpretation allows probability also to be used to express our uncertainty due to our ignorance, known as 'epistemic uncertainty', and popularly expressed as betting odds. Of course there are all gradations, from pure chance (think radioactive decay) to processes assumed to be pure chance (lottery draws), to future events whose odds depend on a mixture of genuine unpredictability and ignorance of the facts (whether Oscar Pistorius will be convicted of murder), to pure epistemic uncertainty (whether Oscar Pistorius knowingly shot his girlfriend).

The judges went on to say:

The chances of something happening in the future may be expressed in terms of percentage. Epidemiological evidence may enable doctors to say that on average smokers increase their risk of lung cancer by X%. But you cannot properly say that there is a 25 per cent chance that something has happened: Hotson v East Berkshire Health Authority [1987] AC 750. Either it has or it has not.

So according to this judgement, it would apparently not be reasonable in a court to talk about the probability of Kate and William's baby being a girl, since that is already decided as true or false (but see note added below). This seems extraordinary.

Part of the problem may be the judges' use of the word 'chance' to describe epistemic uncertainty about whether something has happened or not - this would be unusual usage now (even though Thomas Bayes used 'chance' in this sense). If they had used the term 'probability' perhaps their quote above would seem more clearly unreasonable.

Anyway, I teach the Bayesian approach to post-graduate students attending my 'Applied Bayesian Statistics' course at Cambridge, and so I must now tell them that the entire philosophy behind their course has been declared illegal in the Court of Appeal. I hope they don't mind.

(Note added 1st March 2013: William Hill are currently offering 1000-1 against Chardonnay as the name of the potential future monarch).

Comments

Tom Davis's picture

Tom Davis

Mon, 02/25/2013 - 14:07

Under common law (aka Anglo-American jurisprudence) Rulings of Law must follow precedent. Additionally novel rulings establish precedent. In other words, to be fair to Alice, we must use the same criteria for judgement that we use for Bob. Furthermore, it is established law that someone is "innocent until proven guilty". Using these two axioms, we can see that while refusing to allow bayesian statistics in this case may seem unenlightened, it is nevertheless legally essential that courts do so, as holding someone legally responsible for an action that they did not commit is reprehensible under our legal system, whereas letting someone off the hook for something that they did is merely unfortunate. In our system of law, it is desirable that investigators utilize all available tools, including math, to discover likely culprits and that they use that information to refine and direct their search for actual evidence. It is also desirable that courts refuse to allow the tools to be used as evidence and require that substantial proof be provided.
conjugateprior's picture

conjugateprior

Mon, 02/25/2013 - 15:33

One issue here seems to be the 'closed hypothesis space' that sometimes gets flung at Bayesians, particularly with regard to model checking. Here it seems to come up in point 20 of the orginal report where everything except the cigarette and the arcing theories are ruled out: "The judge rejected the possibilities that the fire was caused by an intruder or by a cigarette discarded by someone other than Mr Nulty. There is no appeal against those conclusions." Not only is there no appeal - that's a legal matter - but the absence of any other possibilities is then built into the problem statement. One might wonder how the (seemingly rather large number of) other possibilities were so speedily assigned prior probability zero, and what might happen if they were not. Indeed, if that conclusion was also itself a 'balance of probabilities' argument then there would seem to be plenty of unacknowledged inferential room to move posterior weight around the hypothesis space - perhaps some of it away from the defence's fag packet.
david's picture

david

Mon, 02/25/2013 - 15:59

I agree. I have never really liked the Sherlock Holmes doctrine as it assumes you have thought of everything. Which is why it is so surprising that this is not the basis that the Court of Appeal use to reject the doctrine: indeed they agree that there are only 3 explanations!

Tony Lloyd's picture

Tony Lloyd

Mon, 02/25/2013 - 17:34

So the Court of Appeal bans Bayesian probability. Personally, I would allow it but bring back forced labour for those Bayesians who cannot substantiate their priors. Of course this might mean that I'm advocating forced labour for Bayesians tout court, but then I'm not entirely convinced that's a bad thing. More seriously, the conclusion (appeal denied, the ciggie explanation is way more likely than the cable arch) sounds compliant with Bayes theorem to me. I think the "past event" comment is really a warning not to confuse epistemic and real world probabilities. The past event either did, or did not happen, and the court has to, at least, have a stab at figuring out if it did. Not to do so risks treating any event with an epistemic probability greater than 50% as "proven on the balance of probabilities" with no further enquiry. This would make for short court cases and reduce the earnings of lawyers (which might be even better than forced labour for Bayesians). But it woudn't seem to serve justice much; I'd hate to be suing an insurance company, walk into court and find the judge actually listening to an argument such as: - In most disputed claims with insurance companies the insurance company turns out to be right, so, on the balance of probabilities, the insurance company is right Of course, the whole thing is written in "lawyer", so we both might be horribly mistranslating it into "normal".
DanielEarwicker's picture

DanielEarwicker

Tue, 02/26/2013 - 09:47

"... forced labour for those Bayesians who cannot substantiate their priors." And an unspecified chance of forced labour for all non-Bayesians, i.e. those who mistakenly believe they have no priors. Ignorantia juris non excusat!
Tom Campbell-Ricketts's picture

Tom Campbell-Ri...

Mon, 02/25/2013 - 19:05

This makes me very angry. Its like the Indiana Pi bill again, only worse. These idiot judges have demonstrated that they haven't the tiniest idea what knowledge actually is, which should scare the bejezus out of everybody. They are not competent to perform their duties, and they should be removed. This is not some trivial choice between one or another statistical philosphy, this is judges rejecting the rational evaluation of evidence.
Paul Hewson's picture

Paul Hewson

Mon, 02/25/2013 - 19:35

Could be misquoted to suggest that the court have just banned something that doesn't exist, like saying Unicorns are not allowed to serve on the jury. Sorry, that doesn't help really does it. More seriously, This ruling bans random effects models used with league tables (and to be fair, we don't teach a lot of people about metapopulations and what not)
nbacchus's picture

nbacchus

Tue, 02/26/2013 - 04:03

It seems more bizarre in light of the fact that DNA evidence is entirely based on Baysian probabilites. I do pity the poor forensic scientists who have to explain this evidence to a jury.
Blaise F Egan's picture

Blaise F Egan

Tue, 02/26/2013 - 09:45

Of the three reasons given for rejecting Holmes' dictum, the first is a point of law and I won't argue with that. The second is right, I think. Conan Doyle could construct situations with only a handful of possibilities but in the real world there are many possibilities, most of them with very low probability of occurrence. In such situations Holmes' dictum is impossible to implement. The third objection seems to me to be a confusion. One starts with an event of a certain probability and then revises it as one considers further evidence. An event that starts off with a low probability can end up as more likely than not, thus meeting the legal need for 'the balance of probability'.
David Cheifetz's picture

David Cheifetz

Tue, 02/26/2013 - 19:40

Dear Sir:

"In a recent judgement the English Court of Appeal has not only rejected the Sherlock Holmes doctrine shown above, but also denied that probability can be used as an expression of uncertainty for events that have either happened or not.

...

So according to this judgement, it would apparently not be reasonable in a court to talk about the probability of Kate and William's baby being a girl, since that is already decided as true or false. This seems extraordinary."

It would be, indeed, were that what the EWCA had stated or implied. But the court did not.

Their Lordships were referring to past events; not future events. What Longmore LJ meant (by the passage you later quoted (from [35] of the reasons) is that in the area of the law involved, once a court decides that a PAST event probably did or did not occur, the occurrence (or non-occurrence) is treated as a certainty.

The point is made, near the end, in the case comment at the second link you provided.

Yours truly,

david's picture

david

Fri, 03/01/2013 - 09:36

I am sorry, I must disagree with this: whether the child is a boy or a girl is a past event that has already been decided - the uncertainty is completely epistemic

Norman Fenton's picture

Norman Fenton

Wed, 02/27/2013 - 16:17

What is interesting about this is that the judge has used almost the same words we have heard from several lawyers. One of the quotes (we gave this in Chapter 1 of our book "Risk Assessment and Decision Analysis with Bayesian Networks" www.bayesianrisk.com) from an eminent lawyer was: “Look the guy either did it or he didn’t do it. If he did then he is 100% guilty and if he didn’t then he is 0% guilty; so giving the chances of guilt as a probability somewhere in between makes no sense and has no place in the law”. Of course, as we show in the book (Chapter 1 is freely available for download) you can actually prove that the judge's assertion is flawed in the sense that it inevitably leads to irrational decision-making. The key point is that there can be as much uncertainty about an event that has yet to happen (e.g. whether or not your friend Naomi will roll a 6 on a die) as one that has happened (e.g. whether or not Naomi did roll a six on the die). If you did not actually see the die rolled in the second case your uncertainty about the outcome is no different than before it was rolled, even though Naomi knows for certain whether or not it was a six (so for her the probability really is either 1 or 0). As you discover information about the event that has happened (for example, if another reliable friend tells you that an even number was rolled) then your uncertainty changes (in this case from 1/6 to 1/3). And that is exactly what is supposed to happen in a court of law where, typically, nobody (other than the defendant) knows for certain whether the defendant committed the crime; in this case it is up to the jury to revise their belief in the probability of guilt as they see evidence during the trial. David points out that the judge is not just 'banning' Bayesian reasoning, but also banning the Sherlock Holmes approach to evidence. But it is even worse, because the judge is essentially banning the entire legal rationale for presenting evidence (namely to determine the probability that the defendant committed the crime). For more on this see: http://www.probabilityandlaw.blogspot.co.uk/2013/02/no-such-thing-as-probability-in-law.html
A.P.Dawid's picture

A.P.Dawid

Sun, 03/03/2013 - 11:45

To follow up Blaise Egon's point about the trial judge's third objection to "Sherlock Holmes' dictum". This was phrased as follows: "If a judge concludes, on a whole series of cogent grounds, that the occurrence of an event is extremely improbable, a finding by him that it is nevertheless more likely to have occurred than not, does not accord with common sense." Which only goes to show how unreliable common sense is as a guide to understanding probability.
Once we appreciate that there is no such thing as "the" probability of an event, but rather a shifting value, relevant to the evidence on which its assessment is based, this apparent paradox is instantly resolved. In this case, two explanations that were both *a priori* unlikely, *before* we learned the fact that a fire occurred, accrue conditional (posterior) probabilities, given that important evidence, that become appreciable. The logic is related to that which *should have been* applied in the Sally Clark case (see http://www.statslab.cam.ac.uk/~apd/SallyClark_report.doc) where, looking forwards, both stories - that her two sons would die of natural causes, or alternatively by murder - were initially extremely unlikely, but, *given that the boys had in fact died*, both stories could be taken seriously. Incidentally, that case also shows that we must be very cautious over applying Sherlock's dictum in a probabilistic setting: a hypothesis that has extremely low *prior* probability should not necessarily be considered impossible. What matters is how its probability compares with those of alternative explanations.

Calum Miller's picture

Calum Miller

Mon, 03/04/2013 - 13:46

Thank you for this piece, David. What a shameful piece of legislation. Any idea how it can be changed?
David Cheifetz's picture

David Cheifetz

Tue, 03/05/2013 - 08:22

Dear Sir: "I am sorry, I must disagree with this: whether the child is a boy or a girl is a past event that has already been decided - the uncertainty is completely epistemic." I am treating this as a response to my comment. With respect, I do not see how it is. Your statement is a true statement; however, I do not see it as a response to anything I wrote. Nor do I see it as responsive to anything in the reasons that you criticize. Let's assume for argument that the world outside the courtroom knows the answer sex of the child. That doesn't mean the tribunal knows what the answer is - if the sex of the child is an issue the court has to decide - unless the evidence required to answer the question is adduced in accordance with the rules of the tribunal. If it isn't, the party with the onus has failed to establish the sex of the child. That's it. Nothing more. Courts make decisions on evidence that is by the rules of the system less than complete. The rules of evidence may (and often do) prevent the judge (or jury) from considering evidence that by a Bayesian (or any other analysis) could be seen as relevant to the decision as to whethe X did or did not occur. Again, all the EWCA meant by the text that you have misunderstood is that once a decision is made about the occurrence or non-occurrence of a past event - even if that decision was based on a 50+ some infinitesimal more % basis - the conclusion is treated as a certainty. The certainty level is no different than if the court were satisfied as to the correctness of on an Ivory Snow basis, or even the basis that Wolfgang Pauli would have used to describe the conclusion opposite to the one the court made. In any event, perhaps you should discuss your understanding of what happened in this particular case with members of the law faculty who teach torts and evidence theory. One of them (should he or she teach torts) might even recognize my name. (Pronunciation may vary). Cheers,
Joe's picture

Joe

Wed, 03/06/2013 - 10:41

I am no statistician, so I could well be wrong. But I think the problem that the judges are pointing at is a logical one: they were offered three alternatives, two were discounted and therefore it was suggested they must accept the third. Quite possibly their explanation of baysian statistics is faulty, I'm not trying to defend the words they used. But the logic seems sound - two options have been disproved, another seems unlikely. But they're not forced to accept that, because there could be some other explanation which hasn't been thought of. I don't think this is an attack on bayesian statistics (and please don't stop teaching it, we need all the statisticians we can get!) unless this situation has occurred before and there is a body of evidence which can be analysed and is reliable enough to make predictions about the future. Holmes is bunk, as was cleverly shown by Chesterton's Father Brown. You are never going to know what you don't know, and hence how can you ever know you've exhausted all possible eventualities? I don't think that is quite the same as using previous observations when you have a lot of the same thing happening. But as I said, I'm no statistician, I could be wide of the mark.
Martin Bland's picture

Martin Bland

Thu, 03/07/2013 - 13:46

A judge banned Bayes' Theorem a couple of years ago: http://www.guardian.co.uk/law/2011/oct/02/formula-justice-bayes-theorem-miscarriage I see David was recruited to oppose this sort of thing. It is not clear from the story where it was the good reverend or the process of estimation with imperfect data that was forbidden.
Dave Marsay's picture

Dave Marsay

Thu, 03/07/2013 - 20:38

I am no lawyer, but my reading of the judgement is that not only should there be 'a balance of probability', but that there should be a preponderance of 'weight of evidence' in order to convict. I would find this reassuring, and far preferable to taking the description too literally. The test might be better expressed as 'on an adequate balance of evidence'. Hard to express without getting mathematical!
Don Mathias's picture

Don Mathias

Sat, 03/09/2013 - 09:38

If I had to argue that the controversial dicta in Nulty v Milton Keynes Borough Council [2013] EWCA Civ 15 at para 37 were correct I would look to see where they came from. The reference is to Hotson v East Berkshire Health Authority [1987] AC 750, and in that case Lord Mackay said: “ ... if a claimant alleges that he sustained a certain fracture in a fall at work and there is evidence that he had indeed fallen at work, but that shortly before he had fallen at home and sustained the fracture, the court would have to determine where the truth lay. If the claimant denied the previous fall, there would be evidence, both for and against the allegation, that he had so fallen. The issue would be resolved on the balance of probabilities. If the court held on that balance that the fracture was sustained at home, there could be no question of saying that since all that had been established was that it was more probable than not that the injury was not work-related, there was a possibility that it was work-related and that this possibility or chance was a proper subject of compensation.” That is, if in Nulty there was evidence that the fire was caused by a stranger or by a rat, there would be no issue of Mr Nulty’s liability. In the case although those possibilities were raised, there was no evidence to support them, so they could be dismissed. But if there had been evidence to support either alternative, the court would have to determine whether it was true. If, on the balance of probabilities the fire had been caused by a stranger, then in this civil case, it had, as a conclusive fact, been caused in that way and the court would not have to move on to consider Mr Nulty’s liability. As a question of “was it or was it not caused by a stranger,” the answer would have been in the affirmative – absolutely and not as a matter of probability. This is what the court must have been thinking of when it said in Nulty at [37], “...But you cannot properly say that there is a 25 per cent chance that something has happened: Hotson v East Berkshire Health Authority [1987] AC 750. Either it has or it has not.” This was said in the context of dealing with the suggested approach at [36] of listing the alternatives and assigning probabilities to each. But alternatives relevant to the live issue (was it Mr Nulty’s cigarette butt or an electrical fault that started the fire) are different to alternatives that avoid the need to consider issue at all. So although the court introduced confusion by failing to specify what sort of issue it was referring to, the dictum is not as alarming for Bayesians as might appear on first reading. This apology for the court is supported by the rest of the judgment. The reasoning is conventional in legal terms, and it is consistent with a Bayesian approach. This is one of several cases where judges seem to be disavowing a mathematical approach while unconsciously obeying mathematical principles.
Don Mathias's picture

Don Mathias

Mon, 03/11/2013 - 03:37

I should add, lest I be accused of sycophantic toadyism, that I think the court’s application of Hotson in para 37 of Nulty is nothing more than a truism. It comes down to saying, when a case is decided, it is decided. This is because in dealing with the issue of alternative cause (for example stranger or rat) the court would have to consider all the relevant evidence, including evidence that the fire was caused by the butt or the electrical fault. The whole case is decided in one trial. The decision process is, or should be, equivalent to multiplying the likelihood ratios for each item of evidence (that is, the probability of getting the evidence on the hypothesis that Mr Nulty caused the fire divided by the probability of getting the evidence on the hypothesis that something else caused the fire), and the priors (the probability that this sort of fire was caused by a cigarette butt, divided by the probability that it was caused by something else), to get the ratio of the probability that Mr Nulty caused the fire to the probability that something else did. Although data for the probabilities would not necessarily be available the court could do its best to make sensible estimates. But statistics should be put before the court wherever possible. So although the court did not reject it, if it had deliberately applied a Bayesian approach it would have avoided its fatuous comment in para 37.
aron's picture

aron

Fri, 03/15/2013 - 03:37

Could somebody explain a "testable" formulation of the concept known as "beyond reasonable doubt" that these lawyer types spake of often without using Baysean Statistics?
Don Mathias's picture

Don Mathias

Sun, 03/17/2013 - 22:34

For a review of the way judges treat "beyond reasonable doubt" see R v Wanhalla, available at http://www.nzlii.org/nz/cases/NZCA/2006/229.html . This is a New Zealand Court of Appeal case but it reviews the law in several other countries. The judgment of Glazebrook J is particularly interesting. I have commented on this case at: http://nzcriminallaw.blogspot.co.nz/2006/08/checking-foundations.html .