Research Methods, Natural Experiments

What is a Natural Experiment?

In natural experiments, which are sometimes called found experiments, researchers do not manipulate the group differences they study. Instead, they take advantage of a naturally occurring change to assess its effects. For example, a preexisting group with access [Page 727]to some form of media (e.g., television, Internet, mobile phone) may be compared with another preexisting group without similar access. Or a preexisting group may be studied before a medium (e.g., television reception) first becomes available and then again after some period of use. Ideally, this group experiencing change in access to the medium of interest will be contrasted with other similar (control) groups whose exposure did not change over the same interval, in a “before and after” study.

Causal Inferences in Natural Experiments

The first condition for making a causal inference, the knowledge that the media cause preceded its effect in time, is usually easily met in natural experiments. This timing difference is probably what interested the researchers in studying the situation.

The second condition for causal inference is that the media exposure (treatment condition) and its effect on users (e.g., on their school achievement, or aggression) must covary. This is usually measured statistically, but statistical errors sometimes occur. For example, there may be a difference in the study's sample that does not exist in the population it represents (often described as a Type I error), or conversely, the statistics used may fail to detect in the sample a true pattern of covariation that does exist in the population (Type II error). Or if there is enough statistical power in a very large sample, then a trivial relationship that is not important either for policy or for theoretical reasons may nevertheless be statistically significant. These three potential false conclusions about covariation include six threats to statistical conclusion validity identified by Cook and colleagues in 1990 and described in detail by MacBeth (1998) in relation to quasi-experiments on the effects of television. One of these six threats, the reliability of measures, applies in natural media experiments primarily to measures of the possible media effects. It applies much less to the measures of media use because one of the advantages of natural experiments is that media exposure is specified as part of the design (e.g., presence versus absence of television or some other medium).

...