Pilot Test

Objectives

In surveys, nonsampling measurement error can be caused by problems associated with interviewers, respondents, and/or the questionnaire. The objective of a pilot test is to identify potential problems and address them prior to the production survey to reduce the amount of nonsampling measurement error produced by the survey.

Procedures

The pilot test procedures should mirror the procedures that will be used in the production survey. For pilot tests, the sample size is typically 50–100 cases. In fact, sample is an inappropriate term to use, since a nonran-dom convenience sample rather than a random sample is typically used. It is important to have a “sample” that is as similar in characteristics as the respondents in the production survey sample. But due to costs and staff efficiencies, pilot tests are frequently done in one to three locations in the country, in particular when data collection is face-to-face, with a relatively small interview staff. If a project involves surveying persons with unique characteristics, it is extremely important that persons with the targeted characteristics be included in the pilot test sample.

All of the procedures used in the production survey should be used in the pilot test. This includes modes of survey administration, respondent rules, interviewer staffing, and interviewer training.

It is not beneficiai to include only the best or most experienced interviewers in the pilot test. Those interviewers often have enough experience that they know how to make a problematic question work, but their solutions sometimes lie outside of standardized interviewing practices and are therefore inconsistent from case to case. Using some inexperienced and/or low-caliber interviewers allows problematic situations to arise naturally and be evaluated. If the situation arises during the pilot test, it is likely it will be encountered in the production survey. It is better to find out about the problem during the pilot test, so that the problem can be addressed prior to production interviewing.

Implementation

When the pilot test is implemented, an estimate of average interview time for questionnaire completion can be obtained that has implications on the survey budget. If the interview time exceeds that which is allowed or budgeted for, then decisions will need to be made about reducing the number of the questions in the survey, reducing the sample size, or changing [Page 584]interview procedures. If there are deep or rare paths of questioning in the questionnaire, reducing the sample size may result in not having enough cases for meaningful analyses of the characteristics associated with the rare paths. The pilot test will also provide information about question comprehension, sensitivity, difficulty, and/or item nonresponse related to specific questions. The pilot test permits issues related to question sequencing and transitional lead-ins to surface. It is important to know whether a lead-in works to segue between questionnaire sections or to serve as a buffer just prior to sensitive questions. Information on case or unit refusals as well as item refusals or other nonresponse is produced through pilot tests. The pilot test provides information about interviewer difficulties related to survey administration and also improvements that may be needed for interviewer training.

...