Quasi Experiments

Validity and Threats to Validity

In 1963, Campbell and Stanley created a validity typology, including threats to validity, to provide a logical and objective way to evaluate the quality of causal inferences made using quasi-experimental designs. The threats are common reasons why researchers may be incorrect about the causal inferences they draw from any cause-probing study, including randomized and quasi experiments. Originally, Campbell and Stanley described only two types of validity: internal validity and external validity. Thomas Cook and Campbell later added statistical conclusion validity and construct validity. We define the validity types shortly. Of the four types of validity, internal validity is the most crucial to the ability to make causal claims from quasi experiments. Internal validity concerns the validity of inferences that the relationship between two variables A and B is causal from A to B. The act of randomization helps reduce the plausibility of many threats to internal validity. Lacking randomization, quasi experiments have to pay particular attention to these threats:

• Ambiguous temporal precedence: the inability to determine which variable occurred first, thereby preventing the researcher from knowing which variable is the cause and which is the effect.
• Selection: systematic differences between unit characteristics in each condition that could affect the outcome.
• History: events that occur simultaneously with the treatment that could affect the outcome.
• Maturation: a natural development over time that could affect the outcome.
• Regression: when units are selected for their extreme scores, they may have less extreme scores on other measures, including later posttests, making it appear as if an effect occurred.
• Attrition: when units who drop out of the one condition are systematically different in their responses than those who drop out of other conditions.
[Page 877]
• Testing: repeatedly exposing units to a test may affect their performance on subsequent tests, appearing as if a treatment effect occurred.
• Instrumentation: changes over time or conditions in the instrument used to measure responses may make it appear as if an effect occurred.
• Additive and interactive threats to internal validity: the impact of a threat can be compounded by, or may depend on the level of, another threat.

The other three types of validity also affect causal conclusions about the treatment and outcome, but they do not necessarily affect quasi experiments more than any other type of experiment. Statistical conclusion validity addresses inferences about whether and how much the presumed cause and effect covary. Examples of threats to statistical conclusion validity are low statistical power and violation of statistical assumptions. Construct validity addresses inferences about higher-order constructs that research operations represent. Examples of threats to construct validity include reactivity to the experimental situation (units respond as they want to be perceived rather than to the intended treatment) and treatment diffusion (the control group learns about and uses the treatment); note that in both these cases, a question is raised about whether the researchers are actually measuring or manipulating what they intended or claimed. External validity addresses inferences about whether a causal relationship holds over variation in persons, settings, treatment variables, and measurement variables. Examples of threats to external validity include interactions of the causal treatment with units or setting, so that the observed causal relationship might not hold in new units or settings.

...