Program Evaluation

Evaluation is the act of judging the merit, worth, value, or significance of things. In terms of out-of-school learning, what is evaluated may be a program, initiative, project, or intervention—for [Page 628]example, an after-school program, a statewide initiative, a robotics project, or a summer learning intervention. Evaluation of out-of-school time (OST) programs provides information for accountability, decision making at multiple levels, better understanding of what works, and demonstrating the achievements of the program.

When evaluation is formally commissioned and conducted by evaluation professionals, it relies on well-developed evaluation questions, systematically collected information (i.e., data), appropriate analytic techniques, and comprehensive, targeted reports to inform stakeholders such as funders, program managers, program participants, and community members of the achievements, challenges, gaps, decisions, participant experiences, and possible directions for the program. This entry further defines evaluation; discusses the differences between program evaluation, monitoring systems, and assessments; and gives an overview of the types of evaluation. It then discusses evaluation in OST programs, how logic models inform program evaluation, and the steps taken to plan and implement program evaluation.

Purposes for evaluation can include (a) accounting to a funder for goals achieved and funds spent; (b) informing decision making and program improvement; (c) understanding the perspectives of beneficiaries of programs, such as youth, families, or communities; and (d) contributing to a general understanding of how programs work by documenting innovations, reach, dissemination, and more. Like strong research, evaluations are guided by overarching questions. These overarching evaluation questions lay the foundation for evaluation designs, methods of data collection, analyses, and reporting. Evaluation uses the same social science methods and analytic procedures as research—surveys, interviews, focus groups, document review, observations, statistical or qualitative data analysis, data interpretation—but for the purposes, as mentioned earlier, of determining merit, worth, value, or significance of a program, initiative, project, or intervention and informing decision making. Many researchers use their skills to conduct evaluations, and many evaluators also conduct research.

Other forms of documenting program activities, such as monitoring systems or assessments, contribute to evaluations, but they are not interchangeable with evaluation. Monitoring systems document, for example, what was done, with whom, for what period of time, in what place, and for how much money. In monitoring systems, activities and participants are counted and described. Assessments ask specific questions about individual-level outcomes such as learning or engagement and are often coordinated with the specific programmatic goals in mind. Both monitoring and assessments contribute to evaluations in that they add data to answer larger questions about the merit, worth, value, or significance of the thing being evaluated. However, in themselves, monitoring systems or assessments (or satisfaction surveys, for that matter) are not evaluation.

An after-school professional development program may use a monitoring system to count the professional development workshops it offered, who and how many after-school professionals attended (out of the total number who were recruited), how many sessions they attended (e.g., four sessions, 3 hours each, 1 each month), and to note that these sessions were offered online and cost the program $500 each to put on. This forms the basis of what is known about the participants in the program and sets the stage for assessment and evaluation. The pre- and postworkshop assessments of skill development (what did the participants learn and have they learned what was expected) are the assessments in this example, and these, together with the monitoring information, contribute to a larger evaluation of the program.

...