Random Order

Random order refers to the randomization of the order in which questions appear in a questionnaire. The purpose is to overcome a type of measurement error known as context effects. This randomization is most often done using a computer program that controls a computer-assisted interview (CAI) being conducted in person, over the phone, or self-administered. Prior to the common use of CAI in survey research, sets of questions were randomized using modified Kish tables that were generated prior to the beginning of the survey and printed on labels indicating to the interviewers what order the questions were to be asked. The order changed randomly for each label to be printed. The labels were pasted next to the question sets in the paper version of the questionnaire. The questions in the set were asked in the random order of the numbers on the label.

There are many issues that impact the response process when a question is presented to a respondent. The items immediately preceding a specific question often have no consequence. Concrete questions addressing facts such as demographic characteristics or behaviors are less likely to be affected by the context effect of the previous questions. However, questions requiring an attitude or opinion can more readily be influenced by the issues addressed in previous questions. Questionnaires are often designed so that a series of topically related attitude questions are presented one after another. In this situation, the likelihood of earlier questions affecting the interpretation of, and responses to, latter questions increases. Presenting the questions in this type of set in a random order minimizes the likelihood of this type of measurement error.

A simple example of this is the set of questions regarding abortion that have been included for many years in the General Social Survey:

• Please tell me whether or not you think it should be possible for a pregnant woman to obtain a legal abortion if:
  • 1.
    The woman's own health is seriously endangered by the pregnancy.
  • 2.
    There is a strong chance of a serious defect in the baby.
  • 3.
    She became pregnant as a result of rape.
  • 4.
    The family has a very low income and cannot afford any more children.
  • 5.
    She is not married and does not want to marry the man.
  • 6.
    She is married and does not want any more children.
  • 7.
    The woman wants it for any reason.

Each of these questions is about abortion, but the scenarios elicit a variety of potential emotions, value judgments, and other unpredictable issues that might influence a person's response to successive questions. There is strong reason to believe that asking the last question (number 7) first each time is likely to result in different percentages of positive and negative responses [Page 682]to questions 1 through 6 than if it were always asked after all the other questions.

This problem is overcome by randomizing the order of the questions in the set for each interview. Current software used in personal (CAPI), self - (CASI), and telephone interviewing (CATI) or in Internet surveying makes implementation of this process quite easy by automatically randomizing question order at presentation but reordering the data for ease of analysis. The resulting aggregate results present a more balanced view than might otherwise be achieved if the questions were always asked in the same order.

...