Interviewer Effects

In many surveys, interviewers play an important role in the data collection process. They can be effective in [Page 370]gaining cooperation of the sample persons, helping clarify survey tasks, or motivating the respondent to provide complete and accurate answers. Thus, interviewers can contribute to data quality, but they can also contribute to measurement error. Interviewers can affect respondents' answers through their mere presence as well as their behaviors when administering the survey.

There are several ways in which interviewers seem to influence respondents' answers. First, the presence of an interviewer can stimulate respondents to take social norms into account when answering a survey question. Pressure to conform to social norms can lead to the underreporting of socially undesirable behavior and the overreporting of socially desirable behavior. Second, observable interviewer characteristics, such as age, gender, or race, can affect many stages of the answer process, for example, by changing the salience of the question topic and therefore altering the retrieval process or by influencing the respondents' judgments of which answers would be socially appropriate. Third, the interviewer's verbal and nonverbal behavior can also affect respondents' answers. For example, the interviewer's feedback, facial expressions, or rate of speech can be taken by respondents as reflecting (dis)-approval of their answers or how important the interviewer thinks the question is. Finally, the interviewer can make errors when delivering and recording the answers to a question. These errors are particularly problematic if they are systematic, for example, not reading certain questions exactly as worded, delivering them incorrectly, omitting necessary probes, or neglecting some response categories.

It is important to note that the effects of interviewers on respondents' answers are not equally strong across all types of questions. Social norms apply only to certain behavioral and attitudinal questions. Interviewers' observable characteristics play a role only if they are related to the question content. Early studies on interviewer effects have shown, for example, race-of-interviewer effects for racial items and gender-of-interviewer effects in gender-related attitude questions but no effects with attitude questions related to other subjects. Similarly, the effects of interviewer behavior also vary by question type. They are more likely to occur if respondents are forced to answer questions about unfamiliar topics, questions about topics that are not salient, questions that are difficult to understand, or questions that leave room for differing interpretations to be elicited by the interviewer. Interviewer errors in question delivery are more likely to occur for longer questions or questions asked in series. Filter questions with long follow-up sequences can provide the opportunity for an interviewer to shorten the questionnaire, even when that is not what the researcher wants to happen.

Interviewer effects can have different consequences for survey estimates. Survey researchers differentiate between systematic interviewer effects that bias survey results and variable interviewer effects that increase the variability of a survey statistic while not introducing bias. Results will be biased if most respondents or certain subgroups systematically deviate in the same direction from the “true” score when interviewed by interviewers with specific characteristics or behavior. Race and gender are examples of such characteristics. But interviewer effects can also increase the variance of a survey statistic without introducing any systematic bias into the estimates. If, for example, interviewers have idiosyncratic ways of phrasing a question or conducting probing, all respondents interviewed by the same interviewer will be affected in the same way, but respondents questioned by another interviewer will be exposed to a different set of idiosyncrasies and might alter their answers in a different way. And even if all the individual biases introduced by the interviewers cancel each other out, the interviewers have the effect of increasing the variance of the respondents' answers. This is usually referred to as interviewer variance.

...