Survey

Surveys as Systematic Data Collections

The idea of systematically studying a population is a first main goal of sample surveys. Surveys therefore typically rely on a standardized questionnaire to gather reliable and valid information from a wide variety of respondents. Reliability, in this context, refers to the idea that the same instrument—applied to comparable samples—will produce consistent results. But reliability is not enough. It is very possible, for example, that a questionnaire consistently measures the wrong construct. Validity therefore adds a second quality criterion and refers to the idea that questionnaires need to provide not just consistent but also unbiased and accurate measurements of people's behaviors, attitudes, and so on. Validity refers to whether a survey measures what its designer thinks it measures.

Reliability and validity are tied to a number of factors in the survey process. But two aspects are particularly important when constructing a questionnaire: the overall structure of the questionnaire and the wording of specific questions.

When structuring a survey questionnaire, the first concern is length. If a survey takes too much time to complete, it will likely result in significant incompletion rates. Unfortunately, the respondents who tend drop out of lengthy surveys are not a random subset of the population. Rather, they tend to be—among other characteristics—younger, more mobile, and employed full time. As a result, excessively long survey instruments often produce samples that are plagued by systematic nonresponse among particular groups in the population and are therefore limited in terms of their generalizability (the results cannot readily be applied to groups other than the one that completed the survey).

A second concern with respect to questionnaire construction is the way questions are ordered on the questionnaire. Well-constructed questionnaires typically ask easy to answer questions first and sensitive or embarrassing questions later in the questionnaire. One of the most common pitfalls in survey instruments involves priming effects (or [Page 854]order effects), entailing the notion that some questions can make certain considerations (for instance, risk or benefits of a specific technology) more salient in a respondent's mind and therefore influence how he or she answers subsequent questions.

Such order effects often emerge when asking respondents whether the risks of a specific technology would outweigh its benefits. First, responses may be biased based on priming or what are sometimes called “response order effects.” Asking respondents first whether the benefits outweigh the risks, followed by response options for “the risks outweighing the benefits,” or “risks and benefits being about equal,” for instance, presents a much different problem than a survey that offers the “risks outweighing the benefits” as the first response option. Acceptable responses tend to be those offered first. Second, this form of measurement forces respondents to make subjective sum-mative judgments about the relative importance of several risks and benefits. Such judgments, unfortunately, are often skewed, given people's tendency to remember unfavorable information about a topic better than favorable information.

...