Standardized Testing and Standards

Test Development

Test development entails many techniques to enable reasonable inferences to be drawn from a sample of individual or institutional performances. Sampling begins by developing a framework for [Page 2074]the “construct” that is to be tested. For example, a framework for the construct of third-grade mathematics might include topics such as addition and subtraction with whole numbers, multiplication, division, and simple fractions. A framework that leaves out important components (such as geometric shapes) or includes irrelevant ones (such as paragraph reading) will threaten the appropriateness of inferences drawn from the test scores.

Once the framework's components have been specified, an appropriate number of test questions (called items) must be developed to sample each component. The number of items is constrained by such things as the length of time the test should take, the amount of money available for test development, and the need for test reliability. When a greater number of items is used to sample each component of the framework, the scores from the test become more reliable or consistent. Reliability is necessary, but not sufficient, for drawing appropriate inferences from scores. For example, a test may yield reliable scores when used over time with similar populations of students. However, if that test included irrelevant components or left out important topics, the score may not permit a valid inference about what the students know and can do. Validity refers to the evidence and argument needed to support the use of a given score for a given purpose.

Test items are analyzed for bias. This is done both through statistical techniques and through the review of items by panels of diverse readers. A finding that different groups of test takers may generate different average scores does not, by itself, indicate a test is biased. Instead, bias refers to inadequacies in the test itself (or in its administration) that create systematic differences in the inferences from scores for different groups of test takers. To illustrate, if average scores from a well-designed and administered standardized test of writing are lower among rural students, this does not mean the test is biased against them. Many test developers consider such score disparities to arise from problems in social policy, not test development or administration.

...