Adaptive and Tailored Testing (including IRT and Non-IRT Application

Introduction

In computerized adaptive testing (CAT), a computer administers the items and gathers the examinee's responses, but its most distinctive feature is that the items finally administered depend on the examinee's ability. The test then adapts to the examinee's performance on the items. The idea of adaptive measurement can be traced back to Binet, but it never became a reality until the appearance of the item response theory (IRT) and the development of the computer. However, adaptive testing is also possible without IRT, as will be seen later. The first ideas on CAT appeared in the early 1970s (Lord, 1970). CAT has spent in the laboratory the greater part of the elapsed time since then, because the main concern of the researchers has been to obtain the most efficient, precise and possible strategies for item selection. They have become operational only in the last decade. Computerized adaptive tests were administered to more than a million people in 1999 (Wainer, 2000). Its main applications are to the areas of personnel selection, educational assessment, certification and licensure. Due to its practical applications, new concerns such as test security, profitability and social impact have arisen.

Basic Principles

The basic principles of a CAT are well established. Its aim is to apply to each examinee only the items that best serve to assess his/her level of ability. Its main advantage is that more efficient measurements are obtained. It needs fewer items (sometimes, less than half) than conventional tests to achieve the same level of precision as a full-length test. The elements that make up a CAT are: an item pool with known properties, a heuristic to choose the items, a method to evaluate ability and a criterion to end the application. Though they are all important, the efficiency of a CAT mostly depends on two closely related complementary processes: the statistical method of estimating ability and the criterion for item selection. This explains the great amount of procedures known and why they are two of the most studied aspects of CAT.

Item Bank

A CAT chooses items from a database (item bank) containing the available items and various information about each item, such as its stem, correct and incorrect options, item parameter estimates under an appropriate IRT model, classical item difficulty and discrimination indices, information on the specific domain the item measures, the proportion of times the item has been administered, etc. The bank has to be calibrated and its unidimensionality and acceptable fitting to an IRT model should be checked and accepted. Item banking and IRT are specific entries in this encyclopedia, and further details can be found there.

A CAT does not need a specific item format. A CAT may be developed both for dichotomous and polytomous items, and for multiple-choice or [Page 10]open-ended items. Items may be visual, auditory and also multimedia items. It is also possible to consider a testlet (cluster of related items) instead of single items as the analysis unit.

An important question to pay attention to is bank size. Well-known high-stake CATs, such as CAT-ASVAB (Sands et al., 1997), have more than one thousand items, but CATs for other uses ordinarily have smaller banks (even below 150 or 200 items). The number of items also depends on the restrictions the item-selection algorithm has implemented and the IRT model in use.

...