Panel Conditioning

Panel conditioning is an effect sometimes observed in repeated surveys when a sample unit's response is influenced by prior interviews or contacts. Various possibilities have been suggested to explain the cause. Panel conditioning can affect the resulting estimates by introducing what is sometimes called “time-in-sample bias” or “rotation group bias.”

In many surveys, the household, business, or other sample unit is contacted more than once over a period of time, usually to reduce the total survey cost, to produce longitudinal estimates, or to decrease the standard error of the estimate of change in the items of interest. In various documented studies, the levels of unemployed persons, expenditures, illness, victimizations, house repairs, and other characteristics were significantly higher or lower in earlier survey contacts than in later ones.

Potential scenarios to explain this behavior are extensive. At times a respondent recalls the answer to a question from a prior interview and repeats it, even when there is a change. Respondents can learn from their past experience answering a questionnaire. In some surveys, certain responses—for example, receiving some type of income or being a victim of a crime—may elicit a lengthy set of follow-up questions or probes. Over time, a respondent may observe this tendency and adjust her or his response to avoid being asked the follow-up sequence. On the other hand, the concepts or questions in a questionnaire may become clearer to a respondent after one or more contacts, producing responses based on better understanding or recall. In these instances, the conditioning can lead to more accurate data.

In surveys that ask for opinions, attitudes, or projected behavior, the person in a sample may become more informed or aware of the issues through a series of interviews. This can affect a later outcome by causing the respondent to explore the topic before the next interview or to change his or her behavior, for example, to vote for a specifie candidate or simply to vote.

The effects of panel conditioning are not always initiated by the respondent. The procedures for conducting a repeated survey can differ from one interview to the next. For example, when the household or business is first contacted, additional relevant questions might be asked that are omitted in later interviews. This omission can influence the subsequent responses. Further, the interviewer may have access to responses from prior interviews and may change the way he or she conducts the next interview based on this information.

In a repeated survey, the effects of panel conditioning on the estimates are difficult to measure and correct for, in part because the effects may be confounded with actual change, panel attrition, the mode of data collection, or other factors. One way to study the effects is to operate a repeated panel simultaneously with independent cross-sections of the same population and compare the results at fixed points in time. Another approach is to compare responses to reliable administrative records and gauge the accuracy over the life of the panel.

Statistically, the group or panel of sample units responding for the first time will exhibit a bias if the mean of their responses differs from the true value based on the entire population. In the same way, the estimate from the panel responding for the second or third time can suffer from bias of a different value. Often this time-in-sample bias is measured by comparing a panel's value to the average over all panels, [Page 567]with the latter as a proxy for the “truth.” However, without additional studies, the true bias of each panel cannot be determined; it can only be expressed relative to a number such as the average.

...