Evaluation: Programme Evaluation (General)

Introduction

The relation between psychological assessment and programme evaluation is a reciprocal one: a great many programme evaluations, especially in the educational and mental health fields, use some kind of psychological test, or other psychological methodology such as focus groups, in order to gather data; and, on the other hand, any systematic use of psychological assessment is a programme and hence a candidate for programme evaluation. For example, the use of test-based, simulation-based, or explicit psychological assessment by a clinician in the hiring or promoting process needs to be, and has occasionally been, evaluated seriously, since the impact on the bottom line (and other matters) is in fact quite variable, despite the intuition of many psychologists that it will be positive. Of course, any other changes in procedures at an organization, such as its treatment of customers, or new hires, or minorities, are also good subjects for evaluation, which often yields surprising and potentially useful results. What follows provides coverage of some of the major developments in the field of programme evaluation across the past few decades, during which the national association (American Evaluation Association) has gone from zero to more than 3000 members, and the number of analogous associations in other countries from zero to about 30. It only sketches the details of the actual process of programme evaluation, which is extremely complex in many cases.

Perspective on Programme Evaluation

Evaluation can be defined, following the dictionaries, as the systematic determination of merit, worth, or significance (hereafter, m/w/s).1 Programmes are just one type of target for this process (all targets for evaluation are known as evaluands; when a person or their work – two different though related matters – is being evaluated, the term evaluee is often used). Programme evaluation is thus best understood as one branch of the applied field of evaluation: other examples of evaluation that are relevant to readers of this work includes personnel assessment itself, product evaluation (e.g. of test instruments, Scantron alternatives, focus group software), policy studies (e.g. of legislation for controlling drug abuse or programmes claimed to reduce obesity), performance evaluation (e.g. the SAT), and proposal evaluation (e.g. at NSF or NIH) are some others. There are in all about 20 named fields of applied evaluation, many of them outside science but entirely disciplined, e.g. the jurisprudence of appellate courts; others are without significant validity, e.g. aesthetic evaluation of modern art; yet others have partial validity, e.g. literary criticism, where in some genres the plot has to avoid inconsistency.

Two branches of applied evaluation are relatively novel as studies, although ancient practices, and of great importance: (i) meta-evaluation [Page 382](the evaluation of evaluations), and (ii) intradisciplinary evaluation (the evaluation of methodological entities within a discipline, e.g. data, experimental designs, interpretations). The first of these is important because it demonstrates the self-referent nature of evaluation and, taken seriously, is the field leading to an answer to The Question That Must Be Answered, namely who evaluates the evaluator? The second is important because it makes a farce out of the doctrine of value-free science, since this kind of evaluation, no different in its logic from any other kind, is part of what every psychologist (and other scientist) has been taught for longer than there has been a doctrine of value-free science. To put it bluntly, the difference between science and pseudo-science is the difference between good evidence, good data, good hypotheses, good inferences, etc., and their bad counterparts, which is of course an evaluative difference. Hence, the idea that there was something scientifically improper about evaluation is absurd.

...