Selection Bias

Selection bias refers to the inaccurate estimates that can occur when a sample of data is nonrandomly selected for study. Samples may be purposefully nonrandomly selected by researchers. More often, however, nonrandom (also called “selective”) samples result from choices made by the actors being studied or by others whose choices affect them. This entry discusses the importance of such biases and the ways to avoid them.

The samples available for studying political processes often are selective. For example, to learn about the escalation or resolution of international disputes, one must study a sample of states involved in disputes. These countries have made or been the recipients of demands or threats. To learn what leads citizens to vote for particular parties or candidates, one must study a sample of persons who have registered to vote and decided to go to the polls. To learn what makes economic sanctions or peace-building missions effective, one must study a sample of cases in which sanctions have been applied or civil wars have ended, respectively. To learn about public opinion, one must study persons who have consented to respond to surveys and to do so informatively. In each of these cases, the sample available for study is not a random sample from the population of interest; instead, it is a sample that has been nonrandomly selected or self-selected.

The economist James Heckman was awarded the Nobel Prize in economics in 2000 for his [Page 2392]work on methods for analyzing selective samples without bias. Heckman popularized these methods and applied them extensively to the study of labor economics. Christopher Achen was the first political scientist to develop methods for analyzing selective samples. Achen explained how nonrandom selection often affects analysts' attempts to evaluate public policy. For example, an analyst might wish to evaluate a pretrial release system: Are appropriate criteria used to decide which defendants will and will not be released? One criterion that judges use in determining whether or not to release a defendant is the seriousness of the accusation; those accused of serious crimes are less likely to be released. The analyst might therefore wish to know whether those accused of serious crimes are more likely to flee or commit additional crimes if released; if not, perhaps judges would want to release more defendants accused of serious crimes. This question is difficult to answer, because the sample of released defendants is selective. When judges do release those accused of serious crimes, it is because they have some additional information (e.g., exemplary courtroom behavior) indicating that these defendants are unlikely to be rearrested. Thus, in the sample of released defendants, everyone is unlikely to commit additional crimes—in some cases because those accused of more minor crimes are unlikely to do so and in some because those who are released after being accused of serious crimes are exceptional. For this reason, a study of released prisoners will show little or no relationship between the seriousness of the accusation and the chance of rearrest, even if such a relationship truly exists. However, methods that correct for selection bias show that those accused of serious crimes are more likely to flee or be rearrested if they are released without regard to courtroom behavior. Failure to correct for selection bias might lead judges to release more defendants accused of serious crimes and to regret this decision when those defendants go on to have a higher rearrest rate.

...