Surveys and Information Bias

Meta-Analyses and Studies

A meta-analysis of 1,607 studies published between 2000 and 2005 in 17 refereed academic journals identified 490 studies that had used surveys. The authors examined response rates from these studies, which covered more than 100,000 organizations and 400,000 individual respondents. The average response rate from individuals was 52.7 percent, with a standard deviation of 20.4, while the average [Page 882]response rate from organizations was 35.7 percent, with a standard deviation of 18.8. The study also found relative stability in response rates in the past decade and higher response rates for journals published in the United States. For organizations, incentives were not found to be related to response rates, while reminders were associated with lower response rates. Electronic data collection via the use of e-mail, phone, or Internet resulted in higher response rates than those obtained through traditional mail methodology.

In another meta-analysis, 59 methodological studies were analyzed to assess the magnitude of nonresponse bias. The strongest predictors were the design features of the surveys, characteristics of the sample, and attributes of the survey statistics computed in the surveys. Other studies have also shown a strong link between nonresponse and measurement errors; in other words, nonrespondents are also likely to be poor respondents if coerced into participation in a survey. For example, past research has demonstrated that nonvoters misreported political views and were less likely than voters to take part in surveys.

Empirical studies have also suggested the use of different recruitment methods for additional survey efforts in order to reduce nonresponse biases. The use of multiple protocols has been shown to increase the response rate, change the point estimates, and achieve lower total nonresponse error. Socioeconomic status also has been shown to influence nonresponse biases—less-educated individuals were shown less likely to have participated in health surveys than those with a higher level of education. A new measure for the risk of nonresponse, the fraction of missing information (FMI), has been proposed as an alternative to the response rate in assessing accuracy of survey results. It measures the level of uncertainty about the values one would impute for current nonresponders and can assist researchers in maximizing the information in the data set.

Survey Research Biases of Consumption and Waste as Determined by Garbology

Since the 1970s, William Rathje and his colleagues have conducted archaeological excavations in several landfills across North America. Their studies produced results that differed from common perceptions as reported via survey data. There was significant difference between what individuals consumed and what they claimed to have consumed. Alcohol consumption was underreported by 40–60 percent, while asparagus consumption was overreported by 200 percent. Other underreported statistics were household wastes, which comprised 15 percent of the foodstuffs; construction debris accounted for 20–30 percent of the materials in the landfills, while paper products took up 40–50 percent.

...