Rubrics

Rubric Development

Rubrics may be characterized as general or task-specific and may use an analytic or holistic scoring system. The purpose of the assessment determines the type of rubric most appropriate for use. For example, if the purpose is to gather information regarding specific components of performance, then a task-specific rubric may be the most appropriate tool. On the other hand, if the purpose is to examine a broader range of tasks, then the evaluator may select a more general rubric for scoring. In much the same way, the scoring system itself can be either analytic or holistic in nature. When [Page 1292]a product or performance comprises a variety of criteria on which judgment is made and each of these criteria is set along a continuum for scoring, the rubric is considered to be analytic. In cases in which only one score is assigned without the emphasis on specific criteria associated with the performance or product, the rubric is considered to be holistic. An example of a commonly used holistic scoring system is the assignment of grades, such as A, B, C, D, and F. Again, the format of the rubric is determined by the purpose of the evaluation.

General guidelines have been established in an effort to assist in the development of scoring rubrics. The first step is to identify the objective of the evaluation. Once this has been determined, the evaluator should identify a series of observable attributes and then thoroughly describe characteristics associated with each. Once the attributes have been identified and the characteristics determined, narratives are written along the continuum (either holistic or analytic), and score points are assigned at each interval. Prior to full implementation of the scoring system, anchors of performance are gathered to ensure that all levels of performance have been identified. Finally, rubrics are revised as needed. The key to the development of any rubric is clearly identifying the goals and objectives to be evaluated. Clear objectives for scoring the product or performance will help determine the format as well as the levels of performance to be scored.

Rubrics can be either too task-specific or too general, having the potential to make scoring difficult. Because rubrics are a measurement tool, attention must be paid to the reliability and validity of the scoring system. Development of a quality rubric is based primarily on the ability of a judge or rater to apply the performance criteria in an objective and consistent manner, ensuring that the scoring criteria adequately measure the objectives of the assessment. The descriptive characteristics of each score point along the scoring continuum may need to be reexamined and sometimes rewritten so that raters can accurately and consistently identify the level of performance based on the descriptors provided. Consistency within and across raters is crucial to the reliability of the rubric. In addition, the degree to which the content or the attributes of performance are aligned with stated goals and objectives should be examined. In some cases, comparisons with existing measures may be useful in helping one determine the quality of the rubric. As with any measurement tool, the overall goal is to collect validity evidence that supports the rubric's ability to accurately measure the underlying psychological construct for which it was intended.

...