Differential Nonresponse

Differential nonresponse refers to survey nonresponse that differs across various groups of interest. For example, for many varied reasons, minority members of the general population, including those who do not speak as their first language the dominant language of the country in which the survey is being conducted, are generally more likely to be nonresponders when sampled for participation in a survey. Thus, their response propensity to cooperate in surveys is lower, on average, than that of whites. The same holds true for the young adult cohort (18–29 years of age) compared to older adults. This holds true in all Western societies where surveys are conducted.

Ultimately, the concern a researcher has about this possible phenomenon should rest on whether there is reason to think that differential nonresponse is related to differential nonresponse error. If it is not, then there is less reason for concern. However, since non-response error in itself is difficult to measure, differential nonresponse error is even more of a challenge.

In considering what a researcher should do about the possibility of differential nonresponse, a researcher has two primary options. First, there are things to do to try to avoid it. Given that noncontacts and refusals are typically the main causes of survey nonresponse, researchers can give explicit thought to the procedures they use to make contact with respondents (e.g. advance letters) and those they use to try to avoid refusals from respondents (e.g. refusal conversation attempts)—in particular as these procedures apply to key groups from whom lower levels of contact and/or cooperation can be expected. For example, the use of differential incentives to persons or households known from past research to be harder to contact and/or gain cooperation from has been shown to be effective in lowering differential nonresponse. However, some have argued that it is not “equitable” to provide higher incentives to groups that traditionally have low response rates because it fails to fairly “reward” those who readily cooperate in surveys.

However, an unpleasant paradox exists for those who argue that differential strategies aimed at reducing differential nonresponse are inequitable to those respondents who are easier to contact and/or more readily cooperate. When a new treatment (e.g. higher noncontingent incentives) is implemented across the board to raise response rates—so that everyone gets the same treatment—it often increases the gap in response rates between the lowest responding groups and the highest responding groups rather than narrowing the gap between the two groups. This results in an increase in the size of the differential nonresponse.

The second option for researchers is to use a variety of post-survey adjustments to their raw data to account for differential nonresponse. If there is no differential nonresponse error associated with the differential non-response, then these adjustments will likely be adequate. However, too often it is not known whether there is any error associated with the differential non-response, and thus researchers cannot know with confidence whether their adjustments have accomplished anything to help make the survey more accurate.