Validation Strategies

Defining the Focus of Validation

It is important to realize that validity is not a characteristic of a test or assessment procedure but of the inferences and decisions made from test or assessment information. Validation is the process of generating and accumulating evidence to support the soundness of the inferences made in a specific situation. Logically, therefore, to examine the concept of validation, it is important to specify (a) the types of inferences involved in applied assessment situations and (b) the nature of evidence that can be used to support such inferences. Different validation strategies reflect different ways to gather and examine the evidence supporting these important inferences.

Applied psychological assessment involves a series of professional activities. A general characterization of these activities includes the following steps: (a) analysis of a work setting to determine (b) the important task and organizational behaviors (and subsequent outcomes) composing a performance domain, which then guide (c) the selection or development of certain assessment procedures (predictors), which make possible (d) predictions about the likelihood that assessees will exhibit important behaviors, and then subsequently (e) measuring individual work behavior using some operational criterion measure. This process implies a conceptual framework, which is presented in Figure 1.

This framework comprises the following inferences:

• Inference 1. The analysis of the work setting yields an accurate description of performance.
• Inference 2. The construct domains tapped by the predictor overlap with the performance domains.
• Inference 3. The predictors are adequate samples of relevant psychological construct domains.
• Inference 4. Predictor scores relate to operational criterion measurements.
• Inference 5. The operational criterion measures adequately sample from the performance domains.
• Inference 6. The predictors relate to the performance domains.

The analysis of a work setting generates a conception of desired performance, or a performance domain. Performance domains are clusters of work activities and outcomes that are especially valued by an organization. Selection decisions based on psychological assessment represent attempts to identify regularities in work behavior—but only those behaviors that are identified by the organization as relevant for goal attainment. Personnel selection, then, is the [Page 860]process of identifying and mapping predictor samples of behavior to effectively overlap with performance domains. Validity can be described as the extent to which the predictor sample meaningfully overlaps with the performance domain.

Figure 1 A Conceptual Framework Detailing the Inferences Involved in Validation

SOURCE: Adapted from Binning and Barrett (1989).

Validation is the process of generating evidence that the inferences drawn from assessment information are accurate. Inference 6 is the keystone inference in applied decision making because it represents whether a specific assessment process yields information that makes possible the prediction of important organizational outcomes. Inference 6 cannot be tested directly because it links predictor measurements with performance domains that are hypothetical domains of idealized work behavior. However, Inference 6 is tied in closed logical loops to other inferences in the framework, and therefore these other inferences play a role in validation. If certain inferences (which will be discussed in the next section) can be substantiated with sufficient evidence, then Inference 6, by implication, is substantiated. To put it another way, validation is the process of generating evidence to support Inference 6, and this involves supporting the other inferences in the framework.

...