Experimental Design

An experiment is a randomized comparison used to assess the effects of a treatment or intervention. In the simplest experiment, participants are assigned at random to one of two groups: One receives the treatment or intervention of interest and the other receives either an alternative treatment or no intervention at all. Both groups of participants are subsequently assessed on an outcome measure, and differences between the groups on the outcome measure are used to estimate the size of the treatment effects.

The alternative to a randomized experiment is a quasiexperiment. A comparison is quasiexperimental if participants are not assigned to treatment conditions at random. For example, a quasiexperiment results when the participants choose for themselves which treatment to receive or when treatments are assigned by others based on criteria such as need, merit, or convenience.

Without random assignment, the participants in the treatment groups can differ in ways that bias estimates of the treatment effect. For example, a medical intervention will look more effective than it is if the intervention is given to those who are the healthiest and the alternative treatment is given to those who are the sickest. In this case, subsequent differences in health would result simply because of the initial differences in health. Random assignment to treatment conditions equates the treatment groups on initial characteristics and thereby avoids bias due to differences in the composition of the groups. As a result, randomized experiments typically produce more valid and precise estimates of treatment effects than do quasiexperiments.

Although random assignment removes bias due to initial group differences, that advantage can be vitiated by differential attrition. Attrition results when not all the participants remain in the study until [Page 150]completion. Differential attrition arises when the participants in the treatment group who leave early differ in important ways from the participants in the comparison group who leave early. Incentives to entice participants to complete the study are often used to reduce biases due to differential attrition.

Randomized experiments are also susceptible to biases from differences in local history. Differences in local history arise when the interventions under study are not the only ways in which the groups of participants are treated differently. For example, if the different interventions are administered at different facilities, those facilities might interact differently with the participants in other ways as well. The best way to avoid local history effects is to control external influences so they are either avoided or distributed equally across the groups.

Randomized experiments are renowned for being difficult to implement. Participants are likely to resist being randomly assigned to treatments they perceive as differing in desirability. Administrators and service providers often prefer to distribute valuable resources based on need, merit, or individual preference rather than by lottery. Also, substantial resources, commitment, and ingenuity are often required to devise mechanisms that can deliver treatments at random.

In spite of their frailties, randomized experiments often remain the preferred method for assessing the effects of treatments when they can be implemented. Quasiexperiments can be far easier to conduct, but biases in quasiexperiments due to initial group differences can be so severe and so difficult to remove through statistical adjustments that well-executed randomized experiments are often the only way to obtain credible assessments of treatment effects.

...