Screening for disease and dishonesty
A secret government agency has developed a scanner which determines whether a person is a terrorist. The scanner is fairly reliable; 95% of all scanned terrorists are identified as terrorists, and 95% of all upstanding citizens are identified as such. An informant tells the agency that exactly one passenger of 100 aboard an aeroplane in which in you are seated is a terrorist. The agency decide to scan each passenger, and the shifty looking man sitting next to you tests positive. Were you sitting next to a terrorist? What are the chances that this man really is a terrorist?
A simple way to untangle this problem is to use natural frequencies; that is, rather than referring to percentages or probabilities, we refer to numbers appropriate for the circumstances. We are given that one of one hundred passengers is a terrorist. When this person is scanned, he or she will probably test positive as a terrorist because of the high accuracy of the scanner. Of the 99 people who are not terrorists, 5% - that's about 5 people - will incorrectly test positive as terrorists. In summary, of the people that test positive, 1 is a terrorist and 5 are not terrorists. So there is only about 1/6 chance of the shifty looking man actually being a terrorist, despite testing positive.
The animation below is used to illustrate this type of problem. Click on the tabs to see different ways of viewing the information. The data shown for 'Security checks' is the data from the problem discussed above. You can insert your own data using the 'Personalise' tab.
The fact that an apparently rather accurate test gives such a low probability of the suspect actually being a terrorist can be surprising. It certainly explains why there may be scepticism about widespread screening for catching criminals: when the proportion of criminality is quite low, even an accurate test will identify many more innocent people than criminals.