Intercoder Reliability Techniques: Fleiss System

Defining Fleiss Method

The element of Fleiss’s kappa, provides a measure of handling the issues of intercoder reliability when more than two coders are employed to evaluate an issue. The type of scale or coding involves the use of nominal or binary rating systems. No version is available for ordinal or interval scale systems, as the general recommendation for that kind of scale would involve the use of a version of Cronbach’s alpha to assess the reliability of the coders.

The general equation for Kappa is the following:

$k=(\mathrm{average}P-\mathrm{average}{P}_e)/(1-\mathrm{average}{P}_e),$

[Page 739]where P indicates the average percentage of observed agreement among the coders and the Pe indicates that average percentage of agreement for each coder due to random chance. The effect of Kappa is to provide an agreement among coders removing or considering the random chance that the two coders would agree. The reliability of the coding scheme depends on coders’ ability to apply the categories in a manner of agreement greater than random chance. This equation is the same as the one used for Cohen’s kappa; the issue is a change, accounting for multiple coders, in providing an estimate of the probability of agreement and random agreement.

To calculate the average percentage agreement, the number of agreements for each category is calculated using the following equation:

$$P=\left\{1/[n*(n-1)]\right\}*\left\{{\displaystyle \sum (n_{ij}{}^2})-n\right\}.$$

This equation indicates the number of possible combinations compared to the number of observed agreements (nij2). This agreement is the agreement across the total number of categories for the particular element or coding scheme. To calculate the average P, the value of P, using this equation for each category in the scheme is used to create an average. For example, if a scheme were to code the clothing of a person in a high school as (a) geek, (b) goth, (c) jock, or (d) tweeker, there would be four different applications of the P equation for each category. The average of P is simply the arithmetic average of the four values for each of the separate categories in the system.

...