Perceived Effectiveness

One goal of formative campaign research is to create or select messages that have the greatest likelihood of influencing the target audience's beliefs, attitudes, or behaviors. Message designers assess the perceived effectiveness (PE) of messages prior to the campaign on the assumption that such measures are indicative of the impact these messages will have during the campaign. Roughly speaking, PE is an estimate of the degree to which the message will produce the intended change in the intended audience. It is frequently assessed using semantic differential scales such as persuasive/not persuasive, effective/ineffective, convincing/not convincing, and compelling/not compelling.

In ideal formative research, message effectiveness would be assessed by directly measuring changes in the attitude, belief, or behavior of the target audience. But, in reality, this is impractical. Persuasive campaigns often encourage the audience to abstain from, rather than engage in, a behavior. For example, the success of an anti-smoking campaign might be measured in terms of the continued absence of the behavior. When the goal of a campaign is slowing or reversing long-term trends in tobacco uptake, observing such change in a small, short-term investigation is highly unlikely. Further, many interventions focus on encouraging actions that are difficult to measure due to concern for privacy. The use of clean needles by intravenous drug users is one example.

In addition, some persuasive messages attempt to frame an issue rather than to call explicitly for enactment of a behavior. An illustration of this can be seen in campaigns that portray tobacco corporations as engaged in active efforts to deceive youth about the effects of smoking. Finally, there is the matter of economics. Evaluating actual effectiveness is an expensive and time-consuming undertaking that is on par with conducting the campaign itself. Jointly, these reasons underscore the need for valid formative research.

Questions concerning the utility of PE can be grouped into two general categories. One involves PE as a mediator between messages features and persuasion outcomes. If PE is genuinely a mediator it must be: (1) sensitive to variations in messages; and (2) shown to predict outcomes of interest such as attitudes, intentions, and behavior. Although little research has addressed the sensitivity issue, there is evidence of statistically reliable associations between message sensation value, self-efficacy, and PE. Studies of the predictive validity are more plentiful. For example, a meta-analysis of 40 effects found a correlation of 0.41 between PE and attitude, a value interpreted as substantial, but imperfect. Experiments focused on the effect of PE on intention provide an indication of causality that is compatible with data from large-scale studies [Page 1049]of antismoking messages. A lagged-correlation analysis of nationally representative survey data demonstrated that judgments of message PE negatively predicted marijuana use one year after message exposure. In short, all of the available data are consistent with treatment of PE as a mediator. Not much is known about factors that influence the magnitude of association at either step in the mediated process.

The second general category of questions focuses on interpretation of mean PE scores. The conceptual problem may be posed as: effective compared to what (or whom)? Existing data suggest that respondents' judgments are likely to be upwardly biased when they construe the question to mean how effective is the message for others rather than how effective is the message for oneself. There is some indication as well that PE judgments may be categorically different, that is, effective versus ineffective, depending on who judges imagine as the audience for the message. One approach to de-biasing the data would be to instruct all participants to make the PE judgment using themselves as the referent. Pragmatically, however, it can be difficult to gather effectiveness data of any sort from the members of certain target audiences. Campaigns to reduce meth-amphetamine use or discourage suicide are two issues that present special problems for formative research. Thus, in principle, average PE judgments may be inflated or depressed depending on the point of comparison used by respondents. It is difficult to assess the frequency or magnitude of this potential problem given the small existing database, but it presents a significant challenge to the application of PE as a formative research tool for health communication campaigns.

...