Response Order Effects

A response order effect occurs when the distribution of responses to a closed-ended survey question is influenced by the order in which the response options are offered to respondents. Primacy and recency effects are two common types of response order effects. Primacy effects occur when response options are more likely to be chosen when presented at the beginning of a list of response options than when presented at the end. In contrast, recency effects occur when response options are more likely to be chosen when presented at the end of a list of response options than when presented at the beginning of the list. The research literature contains myriad examples of both types of effects.

Response order effects are typically measured by presenting different groups of respondents with a survey question with the response options in different orders and assessing the effects of order on the answer respondents give. For example, a random half of respondents in a survey might be asked, Which do you think is more important for success in life: self-control or the ability to enjoy oneself? The other random half of respondents would be asked, Which do you think is more important for success in life: the ability to enjoy oneself or self-control? A primacy effect would be observed if significantly more respondents answered “self-control” in response to the first question than in response to the second, but a recency effect would be observed if more respondents answered “self-control” in response to the second question than in response to the first. In questions with more than two categorical response options, the number of possible response option orders increases dramatically as the number of response options increases (e.g. there are 24 possible response option orders for a question with 4 response options). In questions with response options that fall along a scale (e.g. How likely is it that you will watch the president's speech on television: extremely likely, very likely, somewhat likely, slightly likely, or not at all likely?'), the response options fall into a logical order. For these questions, response order effects can be assessed by providing half of respondents with the response options ordered in one direction (e.g. extremely likely, very likely, somewhat likely, slightly likely, or not at all likely) and providing the other half of respondents with the response options in the opposite direction (e.g. not at all likely, slightly likely, somewhat likely, very likely, or extremely likely).

A number of explanations have been provided for response order effects. For example, some researchers have argued that respondents have difficulty remembering all response options and that response order effects reflect the response options most memorable to respondents (those at the beginning and end of a list of response options). However, researchers have observed response order effects in very simple, short questions with only a few (e.g. two) response options. For these simple questions, it seems unlikely that respondents are unable to recall the question or the response options.

In a large body of evidence regarding response order effects in questions with categorical response options, recency effects have been observed in some cases, primacy effects have been observed in other cases, and in other cases, no significant response order effect was observed. Another theoretical account of response order effects provides an explanation for this mixture of findings. John Krosnick's satisficing theory suggests that although survey researchers hope respondents will answer questions by carefully and thoughtfully going through the four mental processes involved in answering survey questions (i.e. comprehending and interpreting the survey question, retrieving relevant information from memory, integrating that information into a judgment, and mapping their judgment onto the response format provided), respondents may not always be able or motivated to do so. Instead, they may shift their response strategies to minimize effort while providing a “satisfactory” response to the survey question (i.e. satisficing). In doing so, respondents are merely searching for strategies or cues in questions that they can use easily to find a satisfactory answer. One such strategy involves choosing the first response option that seems reasonable, and this strategy is believed to be responsible for response order effects.

...