Test-Retest Reliability

Test-retest reliability is a statistical technique used to estimate components of measurement error by repeating the measurement process on the same subjects, under conditions as similar as possible, and comparing the observations. The term reliability in this context refers to the precision of the measurement (i.e. small variability in the observations that would be made on the same subject on different occasions) but is not concerned with the potential existence of bias.

In the context of surveys, test-retest is usually in the form of an interview-reinterview procedure, where the survey instrument is administered on multiple occasions (usually twice), and the responses on these occasions are compared.

Ideally the reinterview (henceforth referred to as T2) should exactly reproduce the conditions at the original interview (T1). Unfortunately, learning, recall bias, and true changes may have occurred since the original interview, all leading to T2 not matching T1 answers.

Which measurement error components are of interest may affect certain decisions, such as whether to use raw data (if the focus is on response error alone) or edited data and imputed data (if editing and imputation errors are of interest). Oftentimes raw responses need to undergo light editing for practical reasons. Reliability measures (discussed later in this entry) may be calculated for individual questions to identify ones that may be poorly worded or others that may need to be followed by probing. Also, reliability measures may be calculated for population subgroups or by type of interviewer (e.g., experienced vs. novice) to detect problems and improve precision.

Further considerations in the design of a test-retest study include considering whether the interview-reinterview sample should be embedded in the main survey sample or exist as an additional, separate sample. An embedded sample is obviously cheaper, but sometimes a separate sample may be more advantageous. Another issue to consider is using the same interviewer at T2 or not. Using the same interviewer in part of the sample and assigning different interviewers to the rest may help in assessing interviewer effects. The duration of time between T1 and T2 (the test-retest “lag”) often has an effect on the consistency between the interviews. Thus, limiting to a certain time range may be an important consideration.

Furthermore, it is important to properly inform the respondents of the need for the reinterview, because respondents' perceptions of its necessity may affect the quality of the data they provide. After the reinterview is completed, additional special questions may be [Page 889][Page 890]asked of respondents, such as how much they remembered from the original interview and what effect it may have had on the responses. The interviewers themselves may be debriefed on what they observed in the reinterview.