Ranking

There are a variety of formats that can be used in asking survey questions, from items that require a simple “yes” or “no” response to other types of forced-choice items, rating scales, and multi-part items in which respondents' opinions are determined through a series of questions. Ranking is a question response format used when a researcher is interested in establishing some type of priority among a set of objects, whether they be policies, attributes, organizations, individuals, or some other topic or property of interest.

For example, if city officials were interested in whether citizens thought it was most important to improve city services in the area of police protection, fire protection, garbage and trash collection, road maintenance and repair, or parks and recreation, one method they could use would be to use an open-ended question that asked respondents, What service that the city provides do you think is most important to improve in the next 12 months? The percentage of respondents that mentioned each of these services could then be used as a measure of priority for service improvement. Another method for determining this would be to ask a series of rating questions in which respondents were asked whether it was very important, somewhat important, not too important, or not at all important for the city to improve services in each of these five areas, and the option that received the highest percentage of “very important” responses or had the highest average rating would have priority for service improvement. A third way to accomplish this would be through a ranking question, for example: The city provides a number of services, including police protection, fire protection, garbage and trash collection, road maintenance and repair, or parks and recreation. Of these, which do you think is most important for the city to improve in the next 12 months? After respondents selected their “most important” service, they would then be asked, And which service is next most important for the city to[Page 689]improve? three times to identify their second, third, and fourth priority for service improvement, with the service not selected being ranked fifth by default.

Because ranking requires making choices from among a series of options, this format is generally thought to be more difficult (i.e. to create a greater cognitive burden) for respondents than other response formats. This is particularly true for telephone surveys in which the ranking of five items is considered to be the maximum number of objects that the average respondent accurately can rank. Rankings can be used more readily in face-to-face interviews or in a self-administered questionnaire when visual cues (e.g. show cards) can be used to assist respondents in remembering the options and making adjustments in their priority ordering, though there is a limit on the number of ranked items that should be displayed even with these methods of data collection.

When the research question requires a larger number of items to be ranked, several alternatives have been developed. One method is to have respondents rank only those items at each end of their preferences, leaving the middle preferences unranked. For example, if 15 items were to be ranked in terms of their perceived threat to the security of the United States, the three biggest threats could be selected (and these three ranked, if desired, to produce first, second, and third rankings of threat). Similarly, the three least threatening items could be selected (and ranked), producing differentiation at the “biggest” and “least” ends of the scale, but less differentiation in the middle.

...