Design Effects

Design effects stem from the evaluation itself and confound the study results. Design effects can stem from any aspect of the research process, including the measures selected and their reliability or consistency, the validity of each measure (i.e., its ability to measure what it intends to measure), the relevance of each measure to the program's goals, and the feasibility of using the measures as designed within existing evaluation constraints. Stochastic (chance differences) between treatment and control groups will also produce design effects, as will biased study samples in the treatment or control groups that result from refusals, dropouts, and inaccessibility; nonrandom missing data; and reactivity to being in an evaluation.