Generalizability Theory

Partitioning Variance in G-Theory

Within G-theory, variance in scores is typically partitioned through analysis of variance (ANOVA). The type of ANOVA conducted follows from the measurement design, which describes how a given attribute is measured. G-theorists describe measurement designs in terms of facets of measurement—the set of measurement conditions under which data on the objects of measurement (the entities being measured) are gathered. Continuing with the interview example, facets of measurement might include questions, interviewers, and occasions; whereas the objects of measurement would be interviewees. In [Page 267]G-theory, facets and objects of measurement serve as factors in an ANOVA model that is used to generate estimates of their contributions (as well as their interactions' contributions) to variance in scores.

Defining True Variance and Error Variance in G-Theory

Estimates of variance attributable to the object of measurement, facets, and their interactions are often referred to as variance components. The variance component associated with the object of measurement is interpreted as an estimate for true variance—the amount of variability in scores that is attributable to differences between objects of measurement such as interviewees on the attribute of interest including interpersonal skill. G-theorists refer to such variance as universe score variance. Whether a particular variance component is interpreted as error depends on the types of inferences the researcher wants to draw regarding the objects of measurement and the facets of measurement across which the researcher wants to generalize scores.

To illustrate this dependence, consider the interview example discussed earlier. If inferences are restricted to the relative ordering of interviewees on interpersonal skill, only those sources of variance that lead to different orderings of interviewees on interpersonal skill would be defined as error. In G-theory such error is referred to as relative error. Relative error is evidenced by interactions between the objects such as interviewees and facets including questions and interviewers of measurement. For example, the larger the interviewee-by-question interaction, the more the relative ordering of interviewees on interpersonal skill differs depending on the question asked. When error is defined in relative terms, G-coefficients reflect the consistency with which the objects of measurement are ordered on the attribute of interest across facets such as interview questions, interviewers, and occasions. Technically, a G-coefficient is defined as the ratio of universe score variance to universe score variance plus error variance, and it ranges from 0 to 1.

...