Interviewer Variance

Interviewer variance describes the part of the overall variability in a survey statistic that is associated with the interviewer. Clusters of respondents interviewed by the same person tend to have more similar responses than do clusters of respondents interviewed by different interviewers. This cluster effect can appear, for example, if an interviewer uses inappropriate or inconsistent probing techniques, has idiosyncratic interpretations of questions and rewords them accordingly, or differs in the way he or she reads answer categories. In addition, interviewer-specific interactions between the interviewer and respondent can lead to an intra-interviewer covariance term that contributes to the variance of the estimate.

The effect of interviewers on responses can increase the variability of survey estimates in a way parallel to the effect of clustered samples. The standard errors of such survey estimates are inflated compared to those computed for a simple random sample. Thus, ignoring the clustering of respondents within interviewers can yield misleading results in significance tests or in the coverage rates of confidence intervals. Most statistical packages use linearization or replication methods to correct the variance estimation for different kinds of sampling designs. To account for an interviewer clustering effect, those procedures require either an interviewer identification variable or appropriate replicate weights created by the data collector as part of the data set.

The overall variance of the respondent mean is inflated by interviewer variance according to the function deff = 1 + ρ(w − 1), where w is the average number of interviews conducted by individual interviewers, and ρ is the intraclass correlation coefficient among responses produced by a common interviewer. If all respondents interviewed by the same interviewer answered in exactly the same way, ρ would be equal to 1. The size of ρ reported by various researchers has shown substantial variation among surveys and survey variables. The average value for ρ in many (mostly telephone) studies is 0.01, but values of about 0.05 are not uncommon, while for some surveys and items a ρ as high as 0.2 has been observed. These seemingly small values can have a large impact. If the average workload for an interviewer in a survey is 100, a ρ of [Page 385]0.01 can produce a design effect of 1.99. Both a high workload and a high value of ρ contribute to a problematic design effect. A value of deff = 2 would be equivalent to cutting the sample size in half.

Telephone surveys often have high interviewer workloads. Thus, even with low values for ρ, interviewer variance may be a problem and should be accounted for in the data analysis. In face-to-face surveys, not only interviewer variance but a second source of variance contributes to the size of the design effect. Interviewers often work in small geographical clusters to reduce the cost of data collection. The attributes of respondents interviewed by the same interviewer can therefore be correlated simply because people who live in close proximity are likely to be similar to each other in some way. To determine whether it is the interviewer or the geographical proximity that is responsible for the cluster effect, an interpenetrated sample design is required, one in which a random subsample of the full sample is assigned to each interviewer. In practice, there often are considerable limitations to implementing interpenetrated assignments. They are usually too expensive for area probability samples. And randomizing telephone cases among interviewers can usually be done only within the shifts that interviewers work.

...