Quasiexperimental Design

Before-After Comparisons

A before-after comparison is one of the simplest and most common, but also potentially most misleading, designs for estimating a treatment effect. In a before-after comparison, measurements collected before a treatment is introduced are compared to measurements collected after the treatment is introduced. The estimate of the treatment effect is derived from [Page 352]the mean difference between the before and after measurements.

A before-after comparison can be misleading because a mean difference between the before and after measurements can arise from causes other than the treatment and can thereby bias the estimate of the treatment effect. The other causes that can introduce bias are called threats to validity. Seven threats to validity arise most often in before-after comparisons: history, seasonality, maturation, instrumentation, testing, attrition, and statistical regression. Each of these threats to validity is described here as it might apply to a before-after comparison used to assess the effects of a program to help parents manage problem behaviors in their children.

A threat to validity due to history arises when an event other than the treatment occurs between the times of the before and after measurements and causes all or part of the before-after difference. For example, if the program began while school was in session but ended right after spring break, changes in behavior could have resulted because the children went on vacation.

Behaviors can change because of seasonal variations, even in the absence of discrete events such as school breaks and vacations. For example, a change from winter to spring could cause differences in behavior because warmer weather leads to more time spent outdoors. Such an alternative cause is a threat to validity due to seasonality.

A threat to validity due to maturation means there is a trend in the level of the observed outcomes over time due to people getting older, or more tired, or hungrier, or similar factors. For example, some problem behaviors might lessen naturally as children mature.

If the before and after measurements were derived from the parents’ self-reports of their children's troubling behaviors, a threat to validity would arise if the criteria parents used to judge troubling behaviors changed over time. For example, parents might come to realize that behaviors they perceived as troubling at the time of the before assessment were to be expected of all children; by the time of the after assessment, then, the parents would no longer view these behaviors as troubling. Such a change in criteria could make the mean of the before self-reports differ from the mean of the after self-reports even in the absence of a treatment. A change in measurement criteria is called a threat to validity due to instrumentation.

...