Experiments, Field

Definition

Experimentation is a method for obtaining unbiased estimates of causal effects. In social science experiments, the unit of observation is randomly assigned to different interventions, and the effect of the intervention is measured by comparing outcomes of interest across the randomly assigned groups. Field experiments are experiments that take place in real-world settings and attempt to reproduce the environment in which the phenomenon of interest naturally occurs. In contrast, laboratory experiments are often highly stylized representations of the social behaviors the investigators wish to learn about through their experiments, leading to concern that the results from the laboratory will [Page 869]not apply to behavior more generally. The realism of field experiments is intended to minimize concerns about the external validity, or the generalizability, of the experimental results.

Experiments have many components, which may differ in their degree of realism, leading to a blurring of the distinction between what is or is not a field experiment. Economists Glenn Harrison and John List propose a four-category taxonomy to distinguish field experiments from laboratory experiments. According to this taxonomy, a conventional laboratory experiment uses a standard subject pool (i.e., drawn from the university) and an abstract framing of the problem being investigated. Examples of this would be laboratory studies of play in dictator and ultimatum games. Two intermediate categories of experiments, which move in the direction of greater realism, are the artifactual and the framed field experiments. An artifactual field experiment deviates from the conventional laboratory experiment through the use of nonstandard subjects. For example, researchers have investigated ethnic conflict by performing standard laboratory games, such as the dictator game, in Africa, with subjects drawn from groups with a history of ethnic rivalry. The framed field experiment moves further away from the conventional laboratory experiment. This type of experiment uses realistic subjects (as does the artifactual experiment); however, in contrast to the artifactual experiment, the task and context is also more realistic. An example of a framed field experiment in political science is research on the effect of money on legislative access, in which congressional staffers make hypothetical scheduling decisions based on information they are provided about the individual seeking time with the representative. A natural field experiment, which is the design often referred to simply as a “field experiment” in political science, is the same as a framed field experiment, except that the environment is the one where the subjects naturally perform the task in question and the subjects are unaware that they are in an experiment. Research in which political campaigns randomly assign households to receive different campaign mailings to test the effect of alternative communications on voter turnout is an example of a natural field experiment.

This entry focuses on natural field experiments. Although the degree of naturalism in field experiments is the greatest strength of the method, it is important to remember that the goal of experimental interventions is to estimate a causal effect, not to achieve “realism.” If the researcher aims to capture basic psychological processes that do not vary across populations, experimental contexts, or subject awareness of the experiment, then there is no problem with conventional laboratory studies. That said, understanding behavior in natural environments is frequently the ultimate goal of social science research, and it is hard if not impossible to even recognize the full set of threats to external validity present in artificial contexts, let alone adjust the measured experimental effects and measures of uncertainty to account for these threats.

...