Item Order Randomization

Most social scientists are aware that responses to survey questions can be significantly affected not only by how questions are worded but also by the order in which the questions are asked. Furthermore, they are generally aware that the order in which the response alternatives within a question are presented can likewise have a significant influence on survey results. Despite this awareness of order effects in surveys, many investigators either ignore these potential sources of measurement error in designing their questionnaires or fail to systematically control for them by fully randomizing the order in which the items are presented.

Most researchers who suspect there is the potential for an order effect in the questionnaire they are designing will rotate the items, typically presenting them in order X on one form of the questionnaire and order Y on the other. A prototypical example of this practice comes from a November 1997 Gallup poll. On one form of the questionnaire, respondents were asked the following questions in this sequence:

• 1.
  How likely is it, in your view, that a terrorist group will attack the United States using chemical or biological weaponry sometime within the next ten years—very likely, somewhat likely, somewhat unlikely, or very unlikely?
• 2.
  How likely is it, in your view, that a terrorist group will attack the United States using nuclear weaponry sometime within the next ten years—very likely, somewhat likely, somewhat unlikely, or very unlikely?

On the other form of the questionnaire, respondents were asked the same questions in reverse sequence. These alternative sequences produced a significant question order effect. In this case, where there are only two questions to consider, simply reversing the sequence of the items amounts to fully randomizing them and controlling for order effects. But if there are three or more related questions to be asked in a sequence, then full randomization requires more than just asking the questions in one order and the reverse order. With three questions to be asked, for example, there are 3 × 2 × 1, or six possible permutations of the order in which the items can be presented: Q1-Q2-Q3, Q1-Q3-Q2, Q2-Q1-Q3, Q2-Q3-Q1, Q3-Q1-Q2, and Q3-Q2-Q1. It is relatively rare, however, to see such a fully randomized order of item presentation in a survey questionnaire, particularly when the data must be collected using a noncomputerized, paper-and-pencil questionnaire, because of the cost and the impracticality of administering six separate versions of the questionnaire. But fully randomized designs do not appear to be that much more common in computerized telephone and personal interviewing or in Web-based surveys, in which they can be readily implemented by programmers.

Aside from the additional staff time it may take to program such designs, researchers may resist doing so because of the small subsample sizes that will be available to analyze the effects of the different orders of item presentation. For a national sample of 1,000 cases, for example, a fully randomized design of six separate conditions will result in approximately 166 respondents per subgroup, which makes it difficult to detect statistically significant effects and to automatically control for nonrandom measurement errors. These must still be analyzed systematically. Given current practices of simple question or response alternative rotation, many potential order effects in existing data sets have yet to be discovered and controlled for through item order randomization, or even properly understood.

...