Validity: Criterion-Related

Introduction

Criterion-related validity involves verification that a test score is systematically related to an external variable. The current view on validity in the 1999 Standards for Educational and Psychological Testing (AERA, 1999) stresses the unity of psychological theory, evidence, and consequences in supporting the validity of a test used for a particular purpose. Seldom is there a direct use of a test for decision-making, however, that cannot be cast into a framework in which criterion-related validity applies.

For many types of educational and psychological tests, the accurate prediction of a criterion of interest is the primary virtue of the test score. Investigations of criterion-related validity justify using a test for employee selection, college admission, or clinical diagnosis when they demonstrate a correlation between scores on the predictor test and a relevant criterion of success or correct classification and treatment. However, many situations arise in test use in which the relevance of criterion-related validity is less apparent. When an achievement test is used in a high-stakes environment to identify students unprepared for high school, or when a licensure exam is used to deny a teaching credential, the justification for test use is typically based on the relevance of test content to high school coursework or teaching practice. In these settings, poor test performance presumably indicates likely failure in the criterion situation. The difficulty of distinguishing the importance of test content, the statistical relationship of a test to a criterion, and the construct interpretations of both characterizes the unitary view of validity in the 1999 Standards. Thus, the approach to validity described here should be understood in the broader context of current professional standards on validity as a unitary concept.

Studies of criterion-related validity represent one area of test validation research in which quantitative evidence is used to endorse or challenge the legitimacy of actual test use. The combined roles of theory and judgement are critical in forming validity arguments. Yet their influence in actual test use (e.g. individualizing instruction for students in an elementary school) is documented directly in the relation of the test score to an external variable closely matching the desired outcome (e.g. higher achievement from instruction matched to students' developmental level). In this sense, criterion-related validity has been described as an ‘external aspect of construct validity’ (Heubert & Hauser, 1998). This is precisely why criterion-related validity is so important when tests are used to make high-stakes decisions.

The Criterion Problem

Establishing the criterion-related validity of a test presumes the availability of a measurable outcome of interest. A test designed to predict success in a job training programme requires a reliable, valid measure of that success to establish criterion-related validity. This measure could be a supervisor's rating of job performance, a direct measure of productivity such as sales transactions or units assembled, an evaluation of a work sample, or simply the number of days present on the job. Actual measures of criterion performance frequently receive less attention than the tests designed to predict them, rendering the criterion itself the weak link in the argument supporting validity of the predictor.

...