Cohen's Kappa

Calculation

Specifically, k can be calculated using the following equation:

where Po is the proportion of the observed agreement between the two raters, and Pe is the proportion of rater agreement expected by chance alone. A κ of +1 indicates complete agreement, whereas a κ of 0 indicates that there is no agreement between the raters beyond that expected by random guessing or chance alone. A negative κ indicates that the agreement was less than expected by [Page 188]chance, with κ of −1.0 indicating perfect disagreement beyond what would be expected by chance.

To illustrate the use of the equation, let us assume that the results of the assessments made by the two pediatricians in the above-mentioned example are as shown in Table 1. The two raters agreed on the classification of 90 toddlers (i.e., Po is 0.90). To calculate the probability of the expected agreement (Pe), we first calculate the probability that both raters would have classified a toddler as delayed if they were merely randomly classifying toddlers to this category. This could be obtained by multiplying the marginal probabilities of the delayed category, that is, (23÷100)× (27÷100) = 0:062: Similarly, the probability that both raters would have randomly classified a toddler as not delayed is (77÷100) × (73÷100) = 0.562: Therefore, the total agreement expected by chance alone (Pe) is 0.562 + 0.062 = 0.624. Using the equation, κ is equal to 0.73.

Richard Landis and Gary Koch have proposed the following interpretation for estimates of κ (Table 2). Although arbitrary, this classification is widely used in the medical literature. According to this classification, the agreement between the two pediatricians in the above example is “good.”

It should be noted that κ is a summary measure of the agreement between two raters and cannot therefore be used to answer all the possible questions that may arise in a reliability study. For instance, it might be of interest to determine whether disagreement between the two pediatricians in the above example was more likely to occur when diagnosing developmental delay than when diagnosing normal development or vice versa. However, κ cannot be used to address this question, and alternative measures of agreement are needed for that purpose.

...