Confirmation Bias

How Should Inference and Information Acquisition Proceed?

No choice of diagnostic tests can cause confirmation bias if the test results are assimilated in a statistically optimal manner. Therefore, this section first discusses how to incorporate test results in a statistically optimal (Bayesian) way. It then discusses various strategies to select informative tests. Suppose that the base rate of a disease (d) in males is 10% and that a test for this disease is given to males in routine exams. The test has 90% sensitivity (true positive rate): 90% of males who have the disease test positive. Expressed in probabilistic notation, P(pos|d) = 90%. The test has 80% specificity: P(neg|∼d) = 80% (20% false-positive rate), meaning that 80% of males who do not have the disease correctly test negative. Suppose a male has a positive test in routine screening. What is the probability that he has the disease? By Bayes's theorem (see Figure 1, Panel A),

where

Therefore,

Alternately (see Figure 1, Panels B and C), it is possible to count the number of men with the disease and a positive test, and who test positive without having the disease:

where

Figure 1 Different formats for presenting probabilistic information

Notes: The means by which probabilistic information is presented have a large impact on how meaningful the information is to people. The standard probability format (Panel A) is complicated for people to work with, although they can be trained to do so. Both the frequency tree (Panel B) and frequency grid diagram (Panel C) provide more meaningful representations of the information. The term “d” denotes the disease; “∼d” absence of the disease; “p” denotes a positive test. In Panel C, shaded cells denote presence of the disease.

But how should a diagnostic test be chosen in the first place? The fundamental difficulty is that which test is most useful depends on the particular outcome obtained, and the outcome cannot be known in advance. For instance, the presence of a particular gene might definitively predict a disease, but that gene might occur with only one in a million probability. Another test might never definitively predict the disease but might always offer a high degree of certainty about whether the disease is present or not.

Optimal experimental design ideas provide a reasonable framework for calculating which test, on balance, will be most useful. All these ideas are within the realm of Savage's Bayesian decision theory, which defines the subjective expected usefulness (utility) of a test, before that test is conducted, as the average usefulness of all possible test results, weighting each result according to its probability.

...