Generalizability Theory

Generalizability theory provides an extensive conceptual framework and set of statistical machinery for quantifying and explaining the consistencies and inconsistencies in observed scores for objects of measurement. To an extent, the theory can be viewed as an extension of classical test theory through the application of certain ANALYSIS OF VARIANCE (ANOVA) procedures. Classical theory postulates that an observed score can be decomposed into a TRUE SCORE and a single undifferentiated random error term. By contrast, generalizability theory substitutes the notion of universe score for true score and liberalizes classical theory by employing ANOVA methods that allow an investigator to untangle the multiple sources of error that contribute to the undifferentiated error in classical theory.

Perhaps the most important and unique feature of generalizability theory is its conceptual framework, which focuses on certain types of studies and universes. A generalizability (G) study involves the collection of a sample of data from a universe of admissible observations that consists of facets defined by an investigator. Typically, the principal result of a G study is a set of estimated random-effects variance components for the universe of admissible observations. A D (decision) study provides estimates of universe score variance, error variances, certain indexes that are like RELIABILITY COEFFICIENTS, and other statistics for a measurement procedure associated with an investigator-specified universe of generalization.

In univariate generalizability theory, there is only one universe (of generalization) score for each object of measurement. In multivariate generalizability theory, each object of measurement has multiple universe scores. Univariate generalizability theory typically employs random-effects models, whereas MIXED-EFFECTS MODELS are more closely associated with multivariate generalizability theory.

Example

Suppose an investigator, Smith, wants to construct a measurement procedure for evaluating writing proficiency. First, Smith might identify or characterize essay prompts of interest, as well as potential raters. Doing so specifies the facets in the universe of admissible observations. Assume these facets are viewed as infinite and that, in theory, any person p (i.e., object of measurement) might respond to any essay prompt t, which in turn might be evaluated by any rater r. If so, a likely data collection design might be p × t × r, where “×” is read as “crossed with.” The associated linear model is

where the ν are uncorrelated score effects. This model leads to

which is a decomposition of the total observed score variance into seven variance components that are usually estimated using the expected MEAN SQUARES in a random-effects ANOVA.

...