Intercoder Reliability Techniques: Scott’s Pi

When to Use Scott’s Pi

Scott’s pi is only used with nominal level data and can only be applied to situations where there are two coders. If there are more than two coders, Scott’s pi cannot be used, but rather an extension of it known as Fleiss’ kappa can be used. Situations in which there is only one coder do not require a measure of intercoder reliability.

The most common use of Scott’s pi is in textual or content analysis. Scott’s pi allows two coders to go through a text and code for different nominal variables and then to compare their results to determine the extent to which they coded things the same. Having consistency between both coders is essential to content analysis, and Scott’s pi allows a researcher to measure this consistency to detect problems.

The creation of software programs for the coding of text is one area where intercoder reliability can be quite helpful. When designing a program that will analyze and code text according to a set codebook, a researcher wants the computer to be consistent with how it is coding and to ensure that the program is applying the codebook correctly. One way to determine how well a computer program is functioning is to test how the computer and a trained coder both code a data set and then to compare the results. This will allow the researcher to determine whether the program is behaving properly and coding in the same way that a human would.

Strengths and Weaknesses

When attempting to determine how often to expect coders to agree based solely on chance, Scott’s pi takes several factors into account. Scott’s pi takes into account how many different possible categories researchers have, as well as how often each of these categories is used by each coder. This is important because when researchers are developing a codebook, they may have an idea of what they expect to see in the text. Once they actually get into the data and start coding, they may realize that some of the categories they were expecting are simply not there or that there are a few categories that are used significantly more than others. Scott’s pi is able to compensate for this and looks at how often each coder used each category when coming up with an estimate of expected agreement. This makes Scott’s pi a fairly conservative measure of intercoder reliability.

...