Method Variance

A Short History

No measuring instrument is free from error. This is particularly germane in social science research, which relies heavily on self-report instruments. Donald Thomas Campbell was the first to mention the problem of method variance. In 1959, Campbell and Donald W. Fiske described the fallibility inherent in all measures and recommended the use of multiple methods to reduce error. Because no single method can be the gold standard for measurement, they proposed that multiple methods be used to triangulate on the underlying “true” value. The concept was later extended to unobtrusive measures.

Method variance has not been well defined in the literature. The assumption has been that the reader knows what is meant by method variance. It is often described in a roundabout way, in relationship to trait variance. Campbell and Fiske pointed out that there is no fixed demarcation between trait and method. Depending on the goals of a particular research project, a characteristic may be considered either a method or a trait. Researchers have reported the methods that they use as different tests, questionnaires with different types of answers, self-report and peer ratings, clinician reports, or institutional records, to name a few.

In 1950 Campbell differentiated between structured and nonstructured measures, along with those whose intent was disguised, versus measures that were obvious to the test taker. Later Campbell and others described the characteristics associated with unobtrusive methods, such as physical traces and archival records. More recently, Lee Sechrest and colleagues extended this characterization to observable methods.

Others have approached the problem of method from an “itemetric” level, in paper-and-pencil questionnaires. A. Angleitner, O. P. John, and F. Löhr proposed a series of item-level characteristics, including overt reactions, covert reactions, bodily symptoms, wishes and interests, attributes of traits, attitudes and beliefs, biographical facts, others’ reactions, and bizarre items.

Obvious Methods

There appear to be obvious, or manifest, features of measurement, and these include stimulus formats, response formats, response categories, raters, direct rating versus summative scale, whether the stimulus or response is rated, and finally, opaque versus transparent measures. These method characteristics are usually mentioned in articles to describe the methods used. For example, an abs-tract may describe a measure as “a 30-item true–false test with three subscales,” “a structured interview used to collect school characteristics,” or “patient functioning assessed by clinicians using a 5-point scale.”

...