Surveys

Surveys are, arguably, the most popular method for collecting data for evaluations. Surveys involve asking questions of specific individuals and obtaining their responses. The responses are usually tabulated or analyzed and presented as evaluation findings. Conducting a survey is a low-cost and relatively straightforward way to obtain data from many people in a short period of time. However, obtaining answers that accurately reflect the attitudes or behaviors of a study population is more difficult than it may seem on the surface. Fortunately, there has been a great deal of research on survey research methods in the past 20 years that can guide the collection of accurate data.

To develop a survey that will yield an accurate depiction of a study population for an evaluation, evaluators must consider seven important elements of survey design and administration: (a) selecting the individuals to be surveyed, (b) developing the questions for the survey, (c) determining the mode of survey administration, (d) organizing the questions into an instrument, (e) administering the survey, (f) preparing the responses for analysis, and (g) analyzing the data. Each of these elements is briefly discussed in this entry, with attention to some important sources of error and examples from evaluations presented.

The first step in identifying the individuals to be surveyed in an evaluation is to clearly define the target population for the survey; after this, ways to access the study population should be listed and the best way determined. Often evaluators will choose between surveying program participants only and surveying the larger number of individuals who were eligible to participate in the program being evaluated, of whom some participated and some did not. On the one hand, participants can respond to questions about their experiences with the program, such as amount and duration of services received and satisfaction with [Page 403]services. On the other hand, nonparticipants can provide information on coverage rates, services received from other sources, and outcomes in the absence of participating in the program (counterfactuals).

A second consideration in choosing respondents is whether the entire study population will be surveyed or a subset will be selected. Many evaluations, such as evaluations of training workshops, attempt to survey all participants. However, trying to survey all participants is often too costly and actually results in less accuracy than surveying a carefully selected subset of the study population. The source of the inaccuracy is nonresponse, which can bias survey results, because those who choose to respond often have different responses from those who do not.

Selecting a subset of the study population is done through probability sampling methods or nonprobability methods (Henry, 1990). Nonprobability sampling methods, such as purposive sampling or convenience sampling, are often used in evaluations, but the results that are obtained are less likely to reflect the study population accurately. Inaccuracies arise because human judgments are involved in the selection process, such as in the quota sampling methods that led to several newspapers declaring Thomas Dewey as the next president the day after the 1948 election.

Probability samples, which are sometimes called random samples or scientific samples, are defined by methods of selection that result in all members of the study population having a known, nonzero chance of being selected for the sample. Human judgment is removed from the selection process and therefore eliminated as a source of bias or error. Of course, probability samples are subject to nonresponse biases. Careful evaluators allocate resources to reduce nonresponse by following up, often several times, with sampled individuals and persuading them to respond and analyzing to the extent possible between respondents and nonrespondents (Henry, 1990).

...