Impact Assessment

Impact assessment involves a comprehensive evaluation of the long-term effects of an intervention. With regard to some phenomenon of interest, such as a community development program, the evaluator asks: What impact has this program had on the community, in terms of both intended and unintended effects (Patton, 1997)?

Impact assessment brings a broad and holistic perspective to evaluation. The first level of assessment in evaluation is at the resource or input level: To what extent did the project or program achieve the targeted and needed level of resources for implementation? The second level of assessment is implementation analysis: To what extent was the project or program implemented as planned? The third level of analysis focuses on outputs: To what extent did the project or program produce what was targeted in the original plan or proposal in terms of levels of participation, completion, or products produced? The fourth level of analysis involves evaluating outcomes: To what extent and in what ways were the lives of participants in a project or program improved in accordance with specified goals and objectives? Finally, the ultimate level of evaluation is impact assessment: To what extent, if at all, were outcomes sustained or increased over the long term, and what [Page 476]ripple effects, if any, occurred in the larger context (e.g., community)?

Consider an educational program. The first level of evaluation, inputs assessment, involves looking at the extent to which resources are sufficient—adequate buildings, materials, qualified staff, transportation, and so forth. The second level of evaluation, implementation analysis, involves examining the extent to which the expected curriculum is actually taught and whether it is taught in a high-quality way. The third level of evaluation involves outputs, for example, graduation rates; reduced dropout rates; parent involvement indicators; and satisfaction data (student, parents, and teachers). The fourth level, outcomes evaluation, looks at achievement scores, quality of student work produced, and affective and social measures of student growth. Finally, impact assessment looks at the contribution of the education program to students after graduation, over the long term, and the effects on the broader community where the program takes place, for example, community pride, crime rates, home ownership, or businesses attracted to a community because of high-quality schools or repelled because of poorly performing schools.

In essence, then, there are two dimensions to impact assessment, a time dimension and a scope dimension. The time dimension is long term: What effects endure or are enhanced beyond immediate and shorter term outcomes? Outcomes evaluation assesses the direct instrumental linkage between an intervention and participant changes like increased knowledge, competencies, and skills, or changed attitudes and behaviors. In contrast, impact assessment examines the extent to which those outcomes are maintained and sustained over the long term. For example, a school readiness program may aim to prepare young children for their first year of school. The outcome is children's performance during that first year in school. The impact is their performance in subsequent years.

The second dimension of impact assessment is scope. Outcomes evaluation looks at the narrow, specific, and direct linkage between the intervention and the outcome. Impact assessment looks more broadly for ripple effects, unintended consequences, side effects, and contextual effects. Impact assessment includes looking for both positive and negative effects, as well as both expected and unanticipated effects. For example, the desired and measured outcome of a literacy program is that nonliterate adults learn to read. The impacts might include better jobs for those adults, greater civic participation, and even higher home ownership or lower divorce rates. Therefore, impact assessment looks beyond the direct, immediate causal connection to longer term and broader scope effects.

...