Performance Standards: Selected Response Item Formats

Introduction

Setting performance standards means implementing a process that identifies one or more points on a score scale to create categories of observed test scores. More fully, Cizek (1993: 100) has defined standard setting as ‘the proper following of a prescribed, rational system of rules or procedures resulting in the assignment of a number to differentiate between two or more states or degrees of performance’. This differentiation can result in dichotomous classifications such as Master/Non-master or Pass/Fail. Standard setting can also result in more than two categories or achievement levels, such as Basic/Proficient/Advanced or the familiar grades of A, B, C, D, F.

In practice, setting a performance standard has become nearly synonymous with deriving one or more cutting scores. However, as Kane (1994) has pointed out, ‘it is useful to draw a distinction between the passing score, defined as a point on the score scale, and the performance standard, defined as the minimally adequate level of performance for some purpose … The performance standard is the conceptual version of the desired level of competence, and the passing score is the operational version’ (p. 426, emphasis in original).

Specialists in the field of educational measurement have developed numerous methods for deriving levels of performance and a wide variety of applications for standard-setting methods exists. Standards are established for determining school readiness; for communicating student achievement in school subjects; for granting admission to institutions; for selection for special services; for suggesting diagnoses or treatments, for placement into specialized programmes; and for awarding certification or granting licensure. Overviews of the many approaches to standard setting can be found in several sources (see, e.g., Berk, 1986; Cizek, 1996a, 2001; Jaeger, 1989; Livingston & Zieky, 1982).

Although the methods for standard setting are numerous, care must be taken to match the method used to the particular characteristics of the assessment and context in which the standard setting is conducted. Linn (1994) has suggested that standard-setting procedures can be distinguished based on the four unique purposes of exhortation, exemplification, accountability, and certification of achievement. It is also common to categorize methods as reflecting absolute, relative, or compromise standards. Jaeger (1989) has grouped standard-setting methods into two categories, those which are test-centred and those which are examinee-centred.

It can also be useful to classify standard-setting methods by the response format dictated by the items or tasks comprising the assessment, i.e. either constructed-response or selected-response formats. The remainder of this entry consists of two major sections: a review of some of the most common methods for establishing performance standards on selected-response (e.g. multiple-choice) assessments; and a brief summary of professional guidelines for doing so.

Standard-Setting Methods

The following subsections describe major standard-setting methods that have traditionally found wide use on assessments comprised of selected-response format items. The methods are described in order of their introduction in the psychometric literature. All but one of the methods that will be described (Nedelsky) can be – and have been fairly easily adapted to tests consisting of other item/task formats. Conversely, many newly introduced methods have been developed specifically for use with tests consisting of constructed-response items/tasks. Two such examples would be the Bookmark method (Mitzel, Lewis, Patz & Green, 2001) and the Analytic Judgment method (Plake & Hambleton, 2001). It is also important to note, however, that nearly all of the newer methods [Page 691]could be applied to tests consisting of a mix of item formats, or even to tests consisting exclusively of selected-response items.

...