Evaluation, Statistical Challenges in

Challenges for Statistical Evaluation

Lack of a control condition to which to compare the treatment condition is perhaps the number one challenge for statistical evaluation of health campaigns. Without a control group that includes members of the target population who are not exposed to the campaign content, it is impossible to ultimately determine if the campaign itself, or something else, caused observed differences in the group exposed to the campaign. Control groups are often not included as part of evaluation designs due to ethical concerns related to withholding campaign treatments perceived as beneficial or life-changing and due to low resources. [Page 462]Control groups are essential for adding to the evidence base of what campaign strategies actually work, which can allow resources to be funneled to efforts that more effectively facilitate positive health outcomes.

Standard evaluation efforts, if present in a campaign at all, usually consist of a battery of knowledge, attitude, and behavioral intention measures collected shortly after exposure to the campaign. While easy to implement (e.g., a short survey to participants at the site of a program), these measures only tell part of the story of a campaign's success or failure. Demonstrating behavioral effects due to communication efforts usually requires longitudinal designs where the target is evaluated days, weeks, months, and sometimes years after the campaign has been executed. The downside to longitudinal designs, and the reason they are seldom done, is that they are expensive and time consuming. Longitudinal evaluation requires more complicated data collection that includes contacting target audience members repeatedly, often tracking individual-level data, and perhaps conducting more sophisticated data analysis. The benefit of longitudinal designs is that they reveal whether or not the campaign had a sustainable impact on the audience.

Vital for many of the statistical tests used to evaluate health communication efforts (e.g., analysis of variance, or ANOVA, regression), is an assumption of the independence of observations. However, this assumption is hard to ensure because campaign messages and activities can get mixed with other already existing messages or programs intentionally or accidentally. For example, campaign designers might believe they have an adequate control condition for their health campaign, but because participants live and work in two geographically distant places, they could potentially be exposed to the health message at home and discuss it with co-workers who live in the control condition, thereby contaminating the control group and violating the assumption of independence of observations.

...