Handling uncertainty about climate change
The Inter-Academy Council (IAC) recently produced a report on the workings of the Intergovernmental Panel for Climate Change (IPCC) which got a lot of publicity, but almost no coverage was given to what the IAC said about the IPCC's way of handling uncertainty. It makes interesting reading.
In preparation for the Fourth Assessment Report (AR4) of the IPCC in 2007, an attempt was made to standardise the expression of uncertainty across the three Working Groups. Lead authors were advised to consider plausible sources of uncertainty and assess the current level of understanding of key issues using the qualitative scale shown in Table 1.
|Amount of evidence (theory, observations, models)|
|High agreement, limited evidence||.....||High agreement, much evidence|
|Level of agreement or consensus||.....||.....||.....|
|Low agreement, limited evidence||.....||Low agreement, much evidence|
The guidance follows the work of Risbey and Kandlikar in recommending that the precision of any statement about an unknown quantity should depend on the quality of the available evidence, and that numerical probability statements should only be made about well-defined events and when there is ‘high agreement, much evidence’. They distinguish between ‘likelihood’ defined as a ‘probabilistic assessment of some well defined outcome having occurred or occurring in the future’, and ‘levels of confidence’ ‘based on expert judgement as to the correctness of a model, an analysis or a statement’. Tables 2 and 3 provide a mapping between linguistic terms and numerical values for likelihood and confidence .
|Terminology||Likelihood of the occurrence/ outcome|
|Virtually certain||> 99% probability of occurrence|
|Very likely||> 90%|
|About as likely as not||33% to 66% probability|
|Terminology||Degree of confidence in being correct|
|Very high confidence||At least 9 out of 10 chance of being correct|
|High confidence|| About 8 out of 10 chance|
|Medium confidence||About 5 out of 10 chance|
|Low confidence||About 2 out of 10 chance|
|Very low confidence||Less than 1 out of 10 chance|
However, as reported in the critique of the IPCC by the Inter-Academy Council, the Working Groups in the Fourth Assessment Report were not consistent in their use of this guidance. Working Group 1 (The Physical Science Basis) made extensive use of formal models which allowed representations of between-model variability in projecting key quantities as well as overall uncertainties expressed as probability distributions and confidence intervals. Overall conclusions were qualified with a mixture of likelihood and confidence statements, the choice of which sometimes appears somewhat arbitrary: compare “.. very high confidence that the global average net effect of human activities since 1750 has been one of warming” and “Most of the observed increase in global average temperatures since the mid-20th century is very likely due to the observed increase in anthropogenic greenhouse gas concentrations”. Working Group II (Impacts, Adaptation and Vulnerability) primarily used the confidence scale (Table 3), but the IAC report criticised the use of this numerical scale for conclusions that were vaguely worded or based on weak evidence. Working Group III (Mitigation of Climate Change) only used the level-of-understanding scale (Table 1). All Working Groups conditioned on a list of emission scenarios which were not given probabilities.
The IAC concludes that the IPCC Uncertainty Guidance was a good starting point but unnecessary errors arose from its inconsistent use, such as the expression of ‘high confidence in statements for which there is little evidence, such as the widely-quoted statement that agricultural yields in Africa might decline by up to 50 percent by 2020’ (IAC) - which is in any case a fairly vacuous statement. They recommend that future working groups should use the level-of-understanding scale (Table 1) supplemented by quantitative probabilities (Table 2) if there is sufficient evidence, traceable accounts should be provided for expressions of scientific understanding and likelihoods, and that the numerical confidence scale (Table 3) should be abandoned.
Personally, I think the IAC's comments are very appropriate. As argued in the excellent report by Morgan and colleagues on handling uncertainty in climate change, numerical probabilities should be reserved for things that are at least potentially determinable in the future.
I also agree with the IAC that a qualitative assessment of the strength of the underlying science is valuable, but I am not convinced by the scale suggested in Table 1. The world of evidence-based medicine has long wrestled with the problem of distinguishing a numerical estimate of the average effect of a treatment in a population from the quality of the evidence underlying that estimate - for example a large but poorly conducted analysis of a data-base may end up with a tight confidence interval but still not be convincing. The GRADE scale shown in Table 4 is widely used by the Cochrane Collaboration and others as a measure of the quality of the underlying evidence, and provides pragmatic definitions in terms of the possibility of further change.
|Quality of evidence||Definitions|
|High quality||Further research is very unlikely to change our confidence in the estimate of effect|
|Moderate quality||Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate|
|Low quality||Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate|
|Very low quality||Any estimate of effect is very uncertain|
Maybe the IPCC, and other people doing risk assessments on the basis of incomplete information and inadequate knowledge, could learn something from evidence-based medicine?