Validity, External/Internal

External Validity

External validity may be discussed in terms of generalizability of findings of a given study to (a) a specific population (i.e., population validity), (b) across settings or from one environmental condition to another (i.e., ecological validity), or (c) across occasions (i.e., temporal validity). A researcher interested in external validity might ask questions such as these: Can results obtained from a particular sample be replicated with a second sample drawn from the same population? Are results achieved in 4th-grade classrooms generaliz-able to middle-grade students? Are promising results obtained early in the school year generalizable to occasions later in the school year? A common mis-perception among researchers is that evidence of statistical significance equates with evidence of generalizability. Statistical significance only informs about the likelihood of a null hypothesis under the assumption that the sample represents the population. Finding a statistically significant result neither guarantees the goodness of a sample nor makes promises about external validity.

Any factor that challenges the relationship of the findings of a given study to the population(s) or setting(s) of interest is considered a threat to external validity. A researcher might, for example, fail to adequately specify a treatment variable, resulting in the lack of integrity of the treatment. Likewise, unintended treatment interaction effects (e.g., pretest-treatment interactions, multiple-treatment interactions, selection-treatment interactions) can affect participants to the extent that it is difficult to isolate the effects of the treatment. External validity is also threatened by various experimenter effects. For example, high school students involved in a study might react differently to a male versus a female teacher. Further, researchers may unwittingly allow their own attitudes or expectations to affect their interaction with the research participants, resulting in experimenter bias effects. Finally, the way a study is conducted might prompt various unintended reactions from participants, such as the Hawthorne effect, the John Henry effect (compensatory rivalry), or placebo effects, that modify participant behavior and contaminate study outcomes.

Replication of results across multiple studies is crucial in building a case for external validity. [Page 926]However, it is possible to gain preliminary estimates of replicability of results within the limits of a single study by splitting or reconfiguring the sample and recomputing results across these sample subsets using “cross validation,” “jackknife,” or “bootstrap” procedures. Whereas true research replication remains the gold standard for establishing evidence for external validity, use of these sample splitting procedures increases the external validity evidence for a single study. Evidence of external validity is further enhanced when the researcher employs careful description of the sample selection procedures, research methods, and data collection procedures employed. Finally, external validity is enhanced by reporting of statistical effect sizes. These indices provide evidence of strength of findings and extend the usefulness of other commonly used statistical indices such as descriptive statistics and statistical significance test results.

...