Automated Test Assembly Systems

Introduction

Historically, test construction in education and psychology has shown a development from: (1) the construction of standardized tests to the practice of assembling tests from item banks tailored to the test assembler's specifications; (2) the use of intuitive rules of test construction to the application of model-based algorithms; and (3) manual sorting of items on index cards to selection by a computerized system.

Test assembly can be characterized as the task of finding a combination of items from an item pool that satisfies a list of content specifications and is optimal in a statistical sense. Formally, the problem has the structure of a constrained combinatorial optimization problem in which an objective function is maximized subject to a set of constraints, both typically modelled using 0-1 decision variables for the inclusion of the items in the test. Currently, a large variety of test assembly problems have been modelled this way and powerful algorithms for solving them are available.

Modelling Test Assembly Problems

A common view underlying all attempts to automate test assembly is to see each item in the pool as a carrier of a set of attributes relevant to the psychological variable or the domain of knowledge or skills the pool is designed to measure. A formal distinction can be made between the following [Page 124]types of attributes:

• 1
  Categorical attributes, such as item content, cognitive level, format, answer key, and item author. This type of attribute implies a discrete classification of the pool; that is, a partition with classes of items containing the same attribute.
• 2
  Quantitative attributes, such as item parameter estimates, expected response time, previous exposure rate, and word counts. This type of attribute is a value on a variable or parameter that, for all practical purposes, is to be considered as continuous.
• 3
  Logical attributes, which imply relations among subsets of items in the pool, mostly relations of inclusion or exclusion. A relation of inclusions exists if an item has to be presented with other items in the pool because they share a stem or the description of a case. A relation of exclusion exists if items cannot be in the same test form, for instance, because some of them clue the correct answer to the others.

In addition to item attributes, it is useful to introduce the notion of test attributes. A test attribute is defined as a (function on the) distribution of item attributes (van der Linden, 2000a). Examples of test attributes are: the distribution of item content or p-values in a test, its information function, the number of items with a gender orientation, and its (classical) reliability. A test can now be defined as a set of items from a pool that meets a list of specifications with respect to its attributes.

An important distinction is between test specifications formulated as constraints and as objective functions:

• 1
  A specification is a constraint if it requires a test attribute to meet an upper limit, lower limit, or equality.
• 2
  A specification is an objective function if it requires a test attribute to take a minimum or maximum value.

The standard format of a test assembly problem is illustrated by the following example of a classical test assembly

...