How Do and How Should Human Beings Use Reference Classes (Relative Frequencies)?

James Franklin & Scott Sisson, Assessment of Strategies for Evaluating Extreme Risks, (ACERA Project No. 0602, March 2007):

[Previously enumerated] considerations suggest this important conclusion, which is central to the point of view of this report:

It is reasonable to give human intuition the “last word” in risk assessment, while at the same time trying to use formal statistical methods as a kind of prosthesis to supplement its known weaknesses.

A problem where the superiority of human intuition over formal methods is especially evident – and one very relevant to extreme risks – is the “reference class problem” (also called in artificial intelligence “multiple inheritance”). The most basic evidence for probabilities in an individual case is observation of a relative frequency (in a class of which the case is a member). For example, the probability that Tex is rich, given that Tex is a Texan and 90% of Texans are rich, is 0.9. But typically, a case is a member of very many classes, in which relative frequencies vary. And there is no useful theory explaining how to combine the probabilities arising from the different “reference” classes. For example, if the evidence is that Tex is a Texan philosopher, that 90% of Texans are rich and 10% of philosophers are rich, then it is impossible to say how to combine these two numbers to achieve a numerical probability that Tex is rich, on the given evidence. (Hájek, 2006) The problem has caused a great deal of trouble in, for example, the law of evidence, where there is often evidence of different classes but it is of dubious legal relevance (Colyvan et al, 2001; Tillers, 2005), and in attempts to construct medical diagnosis expert systems, where combining evidence from different symptoms is essential but how to do it is theoretically poorly understood. (See also Caponecchia, 2007, section 4 for its relevance to communicating probabilities.)

Yet humans are very good at combining different kinds of evidence. Where they have an advantage over formal methods is that they can learn from long experience the comparative relevance of different reference classes. For example, they can learn enough about being Texan, being a philosopher and being rich to have some sense of whether being Texan or being a philosopher is more likely to be relevant to being rich. The vocabulary of natural languages is already attuned to naming concepts that are relevant to living, that is, are positively relevant in probabilistic inferences; which of them are most relevant to a particular inference is something that itself can be learned – but only over a long period, and in the context of very many other concepts.

That wide base of experience and the resultant tuning of concepts is not something that should be put aside when it comes to extrapolating from experience when evaluating extreme risks. On the contrary, is it a foundation that must be built on. It is the wide base of analogous cases that can compensate for the lack of data of directly relevant cases that is a feature of extreme risk analysis.