External Validity

Two Kinds of Generalizations

Two kinds of generalizations are often of interest to researchers of scientific and scholarly investigations: (a) generalizing research findings to a specific or target population, setting, and time frame; and (b) generalizing findings across populations, settings, and time frames. An example is provided to illustrate the difference between the two kinds. Imagine a new herbal supplement is introduced that is aimed at reducing anxiety in 25-year-old women in the United States. Suppose that a random sample of all 25-year-old women has been drawn that provides a nationally representative sample within known limits of sampling error. Imagine now that the women are randomly assigned to two conditions—one where the women consume the herbal supplement as prescribed, and the other a control group where the women unknowingly consume a sugar pill. The two conditions or groups are equivalent in terms of their representativeness of 25-year-old women. Suppose that after data analysis, the group that consumed the herbal supplement demonstrated lower anxiety than the control group as measured by a paper-and-pencil questionnaire. The investigator can generalize this finding to the average 25-year-old woman in the United States, that is, the target population of the study. Note that this finding can be generalized to the average 25-year-old woman despite possible variations in how differently women in the experimental group reacted to the supplement. For example, a closer analysis of the data might reveal that women in the experimental group who exercised regularly reduced their anxiety more in relation to women who did not; in fact, a closer analysis might reveal that only those women who exercised regularly in addition to taking the supplement reduced their anxiety. In other words, closer data analysis could reveal that the findings do not generalize across all subpopulations of 25-year-old women (e.g., those who do not exercise) even though they do generalize to the overall target population of 25-year-old women.

The distinction between these two kinds of generalizations is useful because generalizing to specific populations is surprisingly more difficult than generalizing across populations because the former typically requires large-scale studies where participants have been selected using formal random sampling procedures. This is rarely achieved in field research, where large-scale studies pose challenges for administering treatment interventions and for high-quality measurement, and participant attrition is liable to occur systematically. Instead, the more common practice is to generalize findings from smaller studies, each with its own sample of convenience or accidental sampling (i.e., a sample that is accrued expediently for the purpose of the research but provides no guarantee that it formally represents a specific target population), across the populations, settings, and time frames associated with the smaller studies. It needs to be noted that individuals in samples of convenience may belong to the target population to which one wishes to generalize findings; however, without formal random sampling, the representativeness of the sample is questionable. According to Thomas Cook and Donald Campbell, an argument can be made for [Page 553]strengthening external validity by means of a greater number of smaller studies with samples of convenience than by a single large study with an initially representative sample. Given the frequency of generalizations across populations, settings, and time frames in relation to target populations, the next section reviews the threats to external validity claims associated with this type of generalization.

...