Check All That Apply

The check-all-that-apply question format presents respondents with multiple response options to a single question, as shown in Figure 1.

In response to the question, the respondents are instructed to select as many of the response options as are perceived to apply to them. Although the check-all-that-apply question format is commonly used in survey questionnaires, research has shown that it can result in a less than optimal response strategy by respondents and may be especially sensitive to primacy effects when the question is asking about past experiences, behaviors, or attitudes.

Figure 1 Check all that apply

When evaluating a list of response options to a check-all-that-apply question, respondents may strive for satis-ficing and burden avoidance. For example, respondents may select only the first of several reasonably acceptable response options and fail to adequately consider the remaining response options before proceeding to the next question. Because of this, some researchers believe it is important to deploy several versions of a check-all-that-apply question, with the response options listed in different orders that are randomly assigned to different respondents, so as to scramble the order of the list of response options across the entire sample.

The check-all-that-apply question format is distinct from the forced choice format (e.g. a list of Yes/No response options). In the forced choice format, respondents are asked to evaluate each forced choice response option individually before moving on to the next. The literature suggests that this difference may result in respondents following divergent cognitive approaches in responding to the forced choice format versus the check-all-that-apply format. In particular, respondents may show more careful consideration and greater cognitive processing of each response option in the forced choice format, while selecting only the first few of several response options that apply in the check-all-that-apply format. Research has shown that in addition to a primacy effect associated with the check-all-that-apply format, the difference between the two formats may result in a higher average number of response options selected per respondent in a forced choice question than in a comparable check-all-that-apply question.

While the addition of the “No” category in the forced choice format should provide greater discrimination when compared to the check-all-that-apply format (which lacks an explicit “No” category), research also has shown that, without adequate instruction, respondents may treat a forced choice format in self-administered questionnaires as Check All That [Page 95]Apply. This occurs when respondents correctly select the “Yes” category for positive responses but fail to select the “No” category for negative responses. As a result, the data can be difficult to interpret. Blank responses may either be intended as a negative response, a not applicable response, or simply an undecided, don't know, or a missing response.

The check-all-that-apply question format is commonly used in self-administered paper-based and Internet surveys. It is less well suited to telephone surveys and consequently is rarely used in that mode. In interviewer-administered in-person surveys, use of the check-all-that-apply format should be paired with the use of a show card displaying the choices to the respondent. In multi-mode surveys, there has been a tendency to pair a check-all-that-apply question in a self-administered questionnaire with a forced choice version in a telephone interview. However, considering the findings in the literature that show that respondents do not treat the two question formats similarly, converting a check-all-that-apply question from a self-administered questionnaire to a forced choice format for use in a telephone interview may not be an optimal approach.

...