Program Evaluation

Program Evaluation

In the private sector, profit is the measurement of success. Very successfulcompanies earn substantial profits. Less successful companies earn less. Failureis rewarded with bankruptcy at best, liquidation of the firm at worst.

The assessment of public-sector organizations and programs is much lessstraightforward. Profit making is not a goal of government programs. Instead,government programs are designed to achieve a plethora of goals, which includeimproving education and human health, protecting the environment, creating jobs,stimulating trade, and increasing transportation safety. In many instances, theintended outcomes of government programs are readily measurable (e.g., a highwaysafety program's primary outcome could be reduced traffic fatalities).Sometimes, the outcomes are much less measurable (e.g., the outcomes associatedwith energy technology research and development [R&D] programs arenotoriously hard to estimate). In all cases, policy makers need to know how costeffective the programs are, how their processes could be improved, and unique tothe public sector, what the programs' impacts on social equity might be.As explained in this chapter, the field of programevaluation emerged to assist government policy makers to measure theoutcomes of government programs, estimate cost effectiveness, and identify waysin which programs could be better implemented and managed.

As employed here, the concept of evaluation refersspecifically to the approaches and methods used to assess public programs thatare provided by government agencies or their contractors. In this context, programs refer to a set of ongoing activities or servicesintended to have some effect on needs, problems, or issues that are the objectof duly enacted public policy and deemed to be in the collective interest.Program evaluation is distinct from basic research, appraisals of individualperformance, monitoring of agency performance, or financial auditing. Forexample, where a program audit typically focuses on whether a program hasoperated in compliance with applicable laws and regulations, program evaluationsemploy social science methods to examine how a program actually functions, whatoutputs are produced, whether desired outcomes are achieved, and whether theprogram itself is responsible for causing some change in an outcome ofinterest.

A common misconception about program evaluation is that it is intended todetermine whether a program succeeded or failed. It is not the purpose of anevaluation to determine whether a program is a success or a failure. Thesejudgments are made by political actors. Evaluations use systematic methods ofobservation and analysis to produce credible findings about program performance,which should inform decisions about the merits ofcontinuing or expanding a program. However, there is no guarantee that suchevidence will figure prominently in a political calculus about resourceallocation. Nonetheless, it is strongly argued that those program leaders andmanagers who have credible and defensible findings from a well-designedevaluation research are not only likely to be better managers but also theycould make a more compelling and competitive case for continued politicalsupport of the program. Conversely, a poorly designed evaluation can misleaddecision makers, waste scarce public resources, and perhaps extend the life of aprogram that poorly serves the intended clientele or exacerbates the conditionsit was intended to alleviate.

Program evaluators find this field extraordinarily challenging. Of course,attention to detail is required to produce well-designed evaluations.Frequently, however, creativity and innovation are also required to devise waysto measure hard-to-measure program outcomes. The purpose of this [Page 407]chapter is to highlight the main issues in evaluation research of whichprogram leaders and managers should be aware and to review the criteria relevantfor judging the adequacy and credibility of evaluation projects.

...