Intercoder Reliability Standards: Stability

Coding

When relying on observation, reliability in coding means that the biases and idiosyncrasies inherent in the observers are substantially less than the “true variation” of the behavior being coded. While a common definition for reliability is the consistency in measures over time, for intercoder reliability, it is further layered to include consistency in the observations between two or more coders—their intercoder reproducibility—in addition to their consistency in coding over time—their intercoder stability—that informs accuracy and data quality.

Such a definition carries the presumption that there is an outcome or behavior that exists independent from the observer. While this assumption is generally accepted in research traditions, such as the experimental worldview, it is not universally accepted in all research traditions. For example, postmodern and standpoint theorists might not accept this assumption and could point to many instances in which characteristics of the observer greatly affected what was observed. As an example, in her research on primates, Donna Haraway details how the very same primate group [Page 733]was observed by an American research group and a Japanese research group. The nature of the observations was quite different, with the Japanese observing much more communal activity than the Americans. Another example, from adolescent health, is the notion of diagnosis. Diagnosing a patient from time 1, to time 2, and so on over time would suggest greater stability in diagnosis and contribute to greater confidence in overall reliability. However, having a common diagnosis from time 1 to time 2, and from multiple researchers/doctors would further enhance overall reliability as it would suggest stability, intercoder stability, and intercoder reliability.

...