Bias

Overview

Survey researchers often rely upon estimates of population statistics of interest derived from sampling the relevant population and gathering data from that sample. To the extent the sample statistic differs from the true value of the population statistic, that difference is the error associated with the sample statistic. If the error of the sample statistic is systematic—that is, the errors from repeated samples using the same survey design do not balance each other out—the sample statistic is said to be biased. Bias is the difference between the average, or expected value, of the sample estimates and the target population's true value for the relevant statistic. If the sample statistic derived from an estimator is more often larger, in repeated samplings, than the target population's true value, then the sample statistic exhibits a positive bias. If the majority of the sample statistics from an estimator are smaller, in repeated samplings, than the target population's true value, then the sample statistic shows a negative bias.

Bias of a survey estimate differs from the error of a survey estimate because the bias of an estimate relates to the systematic and constant error the estimate exhibits in repeated samplings. In other words, simply drawing another sample using the same sample design does not attenuate the bias of the survey estimate. However, drawing another sample in the context of the error of a survey can impact the value of that error across samples.

Graphically, this can be represented by a bull's-eye in which the center of the bull's-eye is the true value of the relevant population statistic and the shots at the target represent the sample estimates of that population statistic. Each shot at the target represents an estimate of the true population value from a sample using the same survey design. For any given sample, the difference between the sample estimate (a shot at the target) and the true value of the population (the bull's-eye) is the error of the sample estimate.

Figure 1 Example of a biased sample statistic

Figure 2 Example of an unbiased sample statistic

Multiple shots at the target are derived from repeated samplings using the same survey design. In each sample, if the estimator of the population statistic generates estimates (or hits on the bull's-eye) that are consistently off center of the target in a systematic way, then the sample statistic is biased.

Figure 1 illustrates estimates of the true value of the population statistic (the center of the bull's-eye), all of which are systematically to the upper right of the true value. The difference between any one of these estimates and the true value of the population statistic (the center of the bull's-eye) is the error of the estimate. The difference between the average value of these estimates and the center of the target (the true value of the population statistic) is the bias of the sample statistic.

...