Information Referenced Testing

A method that has been slow in acceptance by the academic world is an evaluation method called Information Referenced Testing (IRT). Dr. James E. Bruno at the University of California, Los Angeles has been the major proponent of this method. It is not to be confused with Item Response Theory. Item Response Theory is an item analysis method. Although academia has yet to fully embrace this potentially valuable measurement method, industry has been using it for years for purposes of assessment, recertification, and placement. It has had other names, such as admissible probability measurement and probability scoring. Currently it is referred to as confidence-based learning. Its basis is not new. However, its implementation is made feasible by the advent of high-speed, reliable computers. The computations involved with this method are quite extensive and should be done with a computer. Bruno has provided the measurement community with the tools necessary for using this method. He has written a series of computer programs that will enable a person to carry out the computational aspects of this method, as well as two types of answer forms for recording the responses. One of the forms is a simplification of the other.

A staple within the educational and evaluation process is the multiple-choice test. It is simple to implement and score. In fact, the answer sheets designed for this type of testing work very well with the computer technology in scoring. It is considered objective in that the scoring is preset. There is only one right answer to each question. Contrary to this is the essay test, which is subjectively evaluated but at times has the advantage of allowing the student more freedom to show what he or she really knows. The essay test usually has mixed criteria for evaluation and is costly (time-consuming) to grade. The traditional right-wrong multiple choice is a one-dimensional system. Each student's score is compared either to other students' scores or to some criterion of percent correct. In the traditional multiple-choice-type question, a person with partial knowledge may be undecided between two of the choices, let's say choices a and d (and say that choice d is the correct answer). This person has learned enough to know that the other choices (b, c, e) in the question are definitely not correct. However, the student must make one selection only. If she chooses a, she will get the item incorrect. In essence, she would be classified along with the students who picked b, c, or e and suffer the same consequences as they do. Yet she knows more than they do. Wouldn't it be nice if that student could admit that she has only partial knowledge and get to choose both a and d for a small penalty? Wouldn't it be desirable to have a testing system that allows that and more? By more, we mean using a scoring and evaluation system [Page 461]that could tell each student on which items they have full knowledge, partial knowledge, no knowledge, or the wrong knowledge. IRT, when used properly, can provide these. It could also provide for each student an individualized educational plan that tells them where they are weak and where they are strong. (See Table 1.) Additionally, it would also provide teachers with feedback on what the class as a whole knows and doesn't know. This system can provide both a summative and formative evaluation. When we say wrong knowledge, we mean that the person has misinformation. Misinformation can be considered more severe than no information because the person usually has a high level of confidence in the wrong answer. Bruno's research and system has provided educators and students with the means of doing this. IRT is a two-dimensional scoring system in which students are measured against an information standard. Tests using this system are objectively scored, and the student's confidence in his or her answer can be assessed.

...