Program Evaluation

Program Evaluation and Conventional Research

Program evaluation differs from conventional research in a number of ways. Research often investigates how an intervention or product can be generalized to an entire population. Research also aims to identify a causal relation between an intervention or program and the intended effect. Research can also take many years to produce data from which conclusions can be drawn depending on the research questions being addressed. For example, studies focused on the development of a problem are often conducted through longitudinal studies that can last several years.

Program evaluation does not focus directly on the population of a given study, but rather the program or product itself to identify areas for improvement. For example, a program evaluator might investigate if staff training for a program can be done more efficiently, thereby improving the overall functioning of the program. Program evaluation is also interested in whether a program is necessary, thus whether it is worth funding. Consider, for example, a program designed to advocate recycling habits in everyday life. Program evaluators might compare this program with similar programs in aspects such as cost or public reception. If this new recycling program costs more than existing programs but does not produce more instances of recycling behaviors in the population, a program evaluator must ask if this program can be improved or if it is worth continuing. This train of thought leads to another key difference between research and program evaluation: Program evaluation is able to take data as it comes in real time to make adjustments that are believed to improve the performance of a program. If the adjustments improve a program, they can be maintained. If adjustments do not improve a program, they can be dropped for other adjustments that may help.

Despite their differences, program evaluation and research both consider the reliability and validity of their tools and measures. Reliability refers to the extent to which results can be reproduced through repeated trials. Intelligence quotient (IQ) tests are considered reliable when they produce the same results when given more than once to the same person within a short period. Programs and interventions should aim to be just as reliable. Validity refers to an instrument’s ability to accurately measure that which it was designed to measure. If a new inventory was developed to measure IQ, but asked only questions about depressive behaviors, it is likely not a valid test. Both program evaluators and researchers must be aware of the reliability and validity of the tools or measures being used. Whereas researchers might consider [Page 1320]reliability and validity in the early stages of planning their study to find the appropriate measures to use, a program evaluator can decide based on the data at hand whether a program would benefit from a change in design that incorporates more reliable and valid measures.

...