Validity and Reliability

Reliability and validity are two basic concepts in test theory. Reliability refers to the consistency of measures. One type is the test-retest reliability. In this approach, a test is administered to the same group of people twice, with a certain interval between the two administrations. If most examinees get similar test results across the two occasions, evidence is established for high reliability. This type of reliability is also called temporal stability. There are a number of factors that may affect this type of reliability. One factor is the memory effect. If a test is given twice within a short period of time, the examinees will most likely remember their answers to the exam the second time they take the exam. This may result in unusually high correlations between two sets of test scores, hence high reliability. On the other hand, if there is too long a period between two test administrations, students may not remember what they have learned previously. This may lead to lower test scores at the second administration.

The second type of reliability is the alternate-form, or equivalent-form, reliability. It refers to the comparability of test scores one obtains if given two equivalent forms of the same test. Factors affecting this type of reliability include the extent to which the two forms are equivalent and the extent to which the conditions of two administrations are comparable.

The third type of reliability is internal consistency. It refers to the extent to which items on the same test instrument are homogenous with each other. It is distinguished from the previous two types of reliability in that it requires only one test administration. There are a number of formulas used for estimating internal consistency, such as the Cronbach's coefficient alpha and, for dichotomously scored items, the Kuder-Richardson formulas (KR-20 and KR-21). A simpler approach to estimate internal consistency is the split-half method, which divides a test into two halves and correlates the scores on the two to obtain the reliability coefficient.

The fourth type of reliability is the interrater reliability, which refers to the agreement between different raters in their scoring of subjective items for the same examinee. A further type of reliability is intrarater reliability, which refers to the consistency with which a rater applies the same scoring rubrics to score different examinees.

One index for reliability is the reliability coefficient, which is obtained by correlating scores from the same test for a group of examinees. Another index is the standard error of measurement, which is the average difference between the true score and observed scores for an individual. The various types of reliability reflect different sources of error. The generalizability theory uses a single index to take all sources of error into consideration.

Validity refers to the extent to which a test measures what it is supposed to measure and the legitimacy or appropriateness of inferences one makes based on test scores. Traditionally, validity is broken down into content, criterion-related, and construct validity. Content validity refers to the extent to which a test adequately samples the content domain that it tries to measure. The content validity of a fifth-grade math test, for example, may be established if it is determined that what the test covers adequately represents the important mathematical concepts and skills [Page 1048]students learned in that grade. A useful tool is the test blueprint, which is often used during test development to link specific items to important instructional objectives. Expert panel review is often used for standardized tests to ensure content validity.

...