Question Order Effects

The order in which questions are asked in a questionnaire can have a significant effect on the results. The preceding questions provide the context in which the respondent answers an item, and changing this context can make a large difference in the survey results.

There are a number of ways in which items that appear earlier in a questionnaire can affect responses [Page 664]to later questions. One is by establishing a norm of reciprocity or fairness, a frequently cited example of which is provided by the work of Herbert Hyman and Paul Sheatsley from more than 50 years ago. These researchers varied the order of two questions: one on whether the United States should allow communist reporters from other countries to come to the United States and send back to their newspapers the news as they saw it and another on whether a communist country like Russia should let American newspaper reporters come in and send back to America the news as they saw it. Changing the order resulted in a difference of 37 percentage points in the percentage of “yes” responses to the question on communist reporters and a 24 percentage point difference in the results for the American reporters item. When either of the items was asked second, the context for the item was changed as a result of the answer to the first, and the responses to the second were more in line with what would be considered “fair,” based on the previous response.

Another way in which the earlier questions in a survey can affect results is by altering the frame in which a question is interpreted. For example, if respondents are asked about their interest in politics as the first item in a series, their reported level of interest is likely to be higher than if this question appears after a series of potentially difficult political knowledge items, such as whether they happen to remember anything special that their U.S. Representative had done for their district or how they had voted on any particular legislative bill. When the “interest in politics” item is asked first, respondents answer in terms of their general experience and may report a fairly high level of interest based on watching the network news regularly or talking about politics with their friends. Asking the knowledge questions first changes the meaning of “interest in politics”; respondents are more likely to interpret this in terms of having specific knowledge about how their member of Congress has acted. Because most respondents cannot remember any such specific information, the context of the question leads them to report a lower level of political interest.

Question order can also change the salience of various alternatives. In surveys that ask about the most important problem facing the country, for example, this question generally will be one of the first items asked. In this position, respondents are more likely to provide an answer based on their recent experience. If they are first asked a series of questions about some issue such as immigration or public education, the percentage of respondents who cite this issue as “the most important problem” will be higher. By making immigration or public education more salient to the respondent, this question order will change the context in which they interpret the “most important problem” item, and these different question orders will produce different results.

...