Quasi-Experimental Method

Threats to Validity

In 1963, Campbell and Stanley created a validity typology, including threats to validity, to provide a logical and objective way to evaluate the quality of causal inferences made using quasi-experimental designs. The threats are common reasons that explain why researchers may be incorrect about the causal inferences they draw from both randomized and quasi-experiments. Originally, Campbell and Stanley described only two types of validity, internal validity and external validity. Thomas D. Cook and Campbell later added statistical conclusion validity and construct validity.

Of the four types of validity, threats to internal validity are the most crucial to the ability to make causal claims from quasi-experiments, since the act of randomization helps to reduce the plausibility of many internal validity threats. Internal validity addresses whether the observed covariation between two variables is a result of the presumed cause influencing the effect. These internal validity threats include the following:

• Ambiguous temporal precedence: the inability to determine which variable occurred first, thereby preventing the researcher to know which variable is the cause and which is the effect.
• Selection: systematic differences between unit characteristics in each condition that could affect the outcome.
• History: events that occur simultaneously with the treatment that could affect the outcome.
• Maturation: natural development over time that could affect the outcome.
• Regression: occurs when units selected from their extreme scores have less extreme scores on other measures, giving the impression that an effect had occurred.
• Attrition: occurs when those who drop out of the experiment are systematically different in their responses from those who remain.
• Testing: Repeatedly exposing units to a test may permit them to learn the test, giving the impression that a treatment effect had occurred.
• Instrumentation: Changes in the instrument used to measure responses over time or conditions may give the impression that an effect had occurred.
• Additive and interactive threats to internal validity: The impact of a threat can be compounded by, or may depend on the level of, another threat.

The other three types of validity also affect the ability to make causal conclusions between the treatment and outcome but do not necessarily affect quasi-experiments more than any other type of experiment. Statistical conclusion validity addresses inferences about the how well the presumed cause and effect covary. These threats, such as low statistical power and violation of statistical assumptions, are essentially concerned with the statistical relationship between the presumed cause and effect. Construct validity addresses inferences about higher-order constructs that research operations represent. These threats, such as reactivity to the experimental situation (units respond as they want to be perceived rather than to the intended treatment) and treatment diffusion (the control group learns about and uses the treatment), question whether the researchers are actually measuring or manipulating what they intended. External validity addresses inferences about whether the relationship holds over variation in persons, settings, treatment variables, and measurement variables. These threats, such as interactions of the causal treatment with units or setting, determine how well the results of the study can be generalized from the sample to other samples or populations.

...