Krippendorff's Alpha

Reliability Data

Data are considered reliable when researchers have reasons to be confident that their data represent real phenomena in the world outside their project, or are not polluted by circumstances that are extraneous to the process designed to generate them. This confidence erodes with the emergence of disagreements, for example, among human coders regarding how they judge, categorize, or score given units of analysis, in the extreme, when their accounts of what they see or read is random. To establish reliability requires duplications of the data-making efforts by an ideally large number of coders. Figure 1 represents reliability data in their most basic or canonical form, as a matrix of m coders by r units, containing the values ciu assigned [Page 669]by coder i to unit u. The total number of pairable values c is

Figure 1 Canonical Form of Reliability Data

where mu is the number of coders evaluating unit u.

For data to serve reliability assessments, it is necessary, moreover, that (a) units are freely permutable and (b) representative of the data whose reliability is in question; and that (c) coders work independent of each other and (d) must be sufficiently common to be found where the data-making process might be replicated or data are to be added to an existing project.

Alpha

The general form of Krippendorff's α is

where Do is the observed disagreement and De is the disagreement expected when the correlation between the units coded and the values used by coders to describe them is demonstrably absent. This conception of chance is uniquely tied to data-making processes.

When agreement is without exception, Do = 0, α = 1, and data reliability is considered perfect. When Do = De, α = 0, and reliability is considered absent. In statistical reality, α might be negative, leading to these limits:

Small sample sizes might cause the zero value of α to be a mere approximation. The occurrence of systematic disagreement can drive α below zero. The latter should not occur when coders follow the same coding instruction and work independently of each other as is required for generating proper reliability data.

In terms of the reliability data in Figure 1, α is defined—for conceptual clarity expressed here without algebraic

...