Field Experiments

Field experiments are randomized interventions that take place in naturalistic settings, as opposed to research laboratories. Education experiments may take many forms. Examples include preschool readiness programs, curriculum supplements, reductions in classroom size, and alterations in the cooperative format of the classroom, as well as larger institutional interventions, such as voucher systems that allow parents to choose among schools. In each case, field experimentation involves the random assignment of students, classrooms, or schools to treatment and control conditions.

The primary purpose of experimentation is to isolate causal relationships. Random assignment ensures that exposure to the intervention bears no systematic relationship to background factors, such as students' home environment or peer influences. Field settings enable the researcher to draw causal inferences under naturalistic conditions, which enhances the external validity of the results. Field experiments strive to address four aspects of external validity: (1) How closely does the intervention resemble what will be deployed in other settings, for instance, as the result of a new policy initiative? (2) To what extent was the experimental stimulus delivered in a context that resembles the setting within which the intervention is likely to be deployed in the future? (3) How closely do the subjects in the experiment resemble those who are likely to be presented with the intervention? (4) How closely do the outcome measures resemble [Page 401]the outcomes of most interest from a policy or theoretical perspective? The ideal field experiment is one that is conducted as unobtrusively as possible, using subjects and interventions that allow for generalization and outcome measures that meaningfully gauge the short-and long-term effects of the intervention.

Field experimentation is especially useful in educational environments where the intervention and setting interact in complex ways. Here, the advantages over laboratory experimentation or observational research are clear. For example, to test the influence of class size on student performance, researchers in a laboratory study might divide subjects into different sized groups for an afternoon to see how quickly the subjects learn a new skill, such as long division. A researcher conducting an observational (nonrandomized) study of the effects of class size has the ability to observe actual classroom behavior but is unable to distinguish the apparent effects of class size from other factors, such as school funding and parental involvement; even the use of multivariate statistical methods leaves open the possibility that the treatment and control groups differ systematically in unmeasured ways. A field experiment that randomly assigns students to different sized classes during elementary school has the potential benefit of isolating this one change in the students' educational environment while maintaining the advantages of unobtrusive measurement within a naturalistic setting.

In practice, field experiments often confront practical challenges. Schools and parents may be unwilling to participate in a randomized study or to adhere to a protocol that, for example, assigns some students to classes with lower student–teacher ratios. In a class size experiment, for example, parents may withdraw their children from large classes and exit the school or pressure administrators to reassign them to small classes. This problem of noncompliance may be correctable using statistical methods such as instrumental variables regression, where the instrument is assignment to the treatment group and the independent variable is whether or not the subject received the actual treatment. Noncompliance, however, not only complicates the analysis and interpretation of the results; it also undermines the statistical power of the experiment. More difficult to correct statistically are problems of attrition, as occur when subjects assigned to one experimental group are more likely than those in the other group to exit the study. A third problem is spillover, or indirect treatment of some subjects as a function of other subjects receiving a treatment directly. For example, if improving the educational environment among children in small classes has positive educational effects on their counterparts in larger classes, a simple comparison of treatment and control groups will underestimate the effects of the intervention. Finally, field experiments are potentially susceptible to Hawthorne effects. The mere fact that an intervention is administered under the watchful eye of the evaluator may change the quality of the intervention or the manner in which participants respond to it.

...