Unbalanced Question

An unbalanced question is one that has a question stem that does not present the respondent with all reasonably plausible sides of an issue. The issue of balance in a survey question also can apply to the response alternatives that are presented to respondents. Unbalanced questions generally are closed-ended questions, but it is possible to use open-ended questions in which the question stem is unbalanced. An unbalanced question will not always lead to biased data, but that is the concern in most instances.

For example, the following closed-ended question is unbalanced for several reasons and will lead to invalid (biased) data:

First, the question stem is unbalanced because it presents only one side of the issue in noting only one position taken by some in the general public. Second, the response alternatives are not balanced (symmetrical) as there are three “agree” choices and only one “disagree” choice. Third, the three response alternatives have no true midpoint; this is another aspect of the asymmetrical (unbalanced) nature of the response alternatives.

In contrast, a balanced version of this question would be as follows:

This wording is balanced because it poses both sides of the issue. It also has a symmetrical set of response alternatives with two choices for “agree” and two similarly worded choices for “disagree.” Furthermore, it has a true midpoint even though that midpoint does not have an explicit response alternative associated with it. If the researchers wanted to add a fifth choice representing the midpoint they could have added, “Neither Agree nor Disagree” in the middle.

In writing survey questions, it is easy for a researcher to avoid using unbalanced questions, unless there is a specific purpose to use such a question. A [Page 919]legitimate use would be the methodological testing of such question wording to study how such wording affects the answers given by respondents. In some other cases, an unethical or unscrupulous researcher may purposely use unbalanced questions to bias data in the directions that favor his or her client's interests. For example, a client with a particular political agenda can have a researcher word questions in an unbalanced fashion to make it more likely the answers respondents give are in a direction favored by the client. The unbalanced question—Many people believe that American troops should be withdrawn from Iraq as soon as possible. Do you Strongly Agree, Agree, Somewhat Agree, or Strongly Disagree?—makes it more likely that respondents will agree than if they were presented with a balanced version of this question. If the client wanted data to show that a larger proportion of the public wants troops to be withdrawn, then using an unbalanced question such as this would accomplish that end. Because too few reporters, editors, and producers in the news media ask to see the exact question wording that underlies survey data, they are not in a position to know whether the data are likely to be biased. As a result, data from unbalanced questions are easily disseminated without the news organization or the public knowing better.

...