List-Experiment Technique

The list-experiment technique is a survey measurement technique that uses an experimental design to [Page 433]measure a sensitive topic in a way that circumvents much of the questionnaire-related and respondent-related measurement errors that may result from using other methods due to the biasing effects of social desirability and privacy concerns. For example, in 2007, a study using the list-experiment technique reported that more than one fourth of Americans (26%) would be “upset or angry” if the country were to elect a female president. This percentage was much higher than what had been reported previously about the magnitude of this concern from surveys that used more standard (nonexperimental) measurement techniques.

In its simplest form, the list-experiment randomly assigns a sample of respondents to one of two groups: a control group and a treatment group. The control group is exposed to a questionnaire version in which they are asked to consider a list of statements that may or may not apply to them; then they are asked to report “how many” of the statements apply. They are not asked to indicate which of the statements in the list apply, but merely to indicate the total number of statements that apply. For example, if there are four statements, then a control group respondent merely provides an answer from “zero” to “four.” In contrast, the treatment group is exposed to a different version of the questionnaire in which they are asked to consider the same statements, but the list they are given includes one more statement than the list given to the control group. Neither the control group nor the treatment group is aware that they are being exposed to a different list. The treatment group is not asked to indicate which statements apply but merely to give the total number of statements that apply. If, for example, the control group is given four statements, then the treatment group is given a list with the same four statements plus one additional (fifth) statement. This additional statement is about the sensitive issue that the researchers are trying to measure accurately.

By comparing the average answer given by the respondents in the treatment group with the average answer given by the control group respondents, the researchers have an unobtrusive way of estimating how many people in the target population the sensitive issue applies to. Because this technique is used when a controversial or otherwise sensitive issue is being measured, there are reasons for the researchers to be concerned if the issue were asked about directly. The concern is that if that were to happen, a reduced proportion of respondents (i.e. biased on the low side) would be identified as having the issue applying to them.

For illustration purposes, imagine that a researcher was interested in measuring whether or not a person had a handgun in his or her home. Asking this question directly in a survey would likely yield biased data because of social desirability and privacy effects. Using the list-experiment technique, the researcher would randomly assign respondents to one of two versions of the questionnaire. One version, the control condition, could contain the list of items shown in Table 1. The other version, the treatment condition, could contain the list of items shown in Table 2. The sensitive item added to the list in Table 2 is I have a handgun at home.

...