Evaluation Research

Categories of Evaluation Research

Evaluation research generally falls into two broad categories: (1) formative or process-related evaluation and (2) summative or outcome-related evaluation. In practice however, it is common to have some questions that are formative in nature and others that are summative within the same study. Formative evaluations are designed to answer questions about how a program works. A major goal of formative evaluations is to support midcourse corrections and improve the program. Summative evaluations, in contrast, provide information on the program’s efficacy (outcomes when implemented under ideal conditions) or effectiveness (outcomes under routine conditions). It is important to note that summative findings are strengthened if they hold up under repeated testing, under different conditions, and with different populations.

Evaluators need to be familiar with a range of designs, as different programs may require different combinations of design elements. Some evaluation research designs are particularly helpful in the early developmental stages of a program, whereas others are more suited to situations in which the program has become more routinized and regularly implemented. For example, in the early stages of creating a program, the goal is likely to be developing or improving the program and working out implementation challenges. Thus, a formative evaluation using descriptive design elements would be appropriate. Conversely, the evaluation of a stable and well-defined program is likely to be geared toward estimating its impact. Here, a summative evaluation using experimental or quasi-experimental designs would be more appropriate.

While no single design can evaluate the wide variety of possible programs and their complexities, the What Works Clearinghouse (WWC) has developed a set of evidence standards for judging the rigor of designs, which unequivocally elevates the experimental design as the gold standard. Established by the U.S. Department of Education in 2002 to be a source of scientific evidence for what works in education, the WWC recognizes that randomized experiments are not always possible, and so rigorous quasi-experimental designs (QED) are also acceptable but are given a lower rating. Next, descriptive, quasi-experimental and experimental designs are described in more detail.

...