Intercoder Reliability Techniques: Scott’s Pi

Scott’s pi is a measure of intercoder reliability for nominal level data with two coders. Scott’s pi was [Page 753]developed by William A. Scott in 1955. The formula for Scott’s pi is:

$$\pi =\mathrm{Pr}(a)-\mathrm{Pr}(e)1-\mathrm{Pr}(e).$$

Pr(a) represents the amount of agreement that was observed between the two coders. Pr(e) represents the amount of agreement that is expected between the two coders. Scott’s pi works by comparing the amount of agreement observed between the two coders with how much agreement is expected if both coders chose randomly. If both coders are consistent, the amount of observed agreement will be higher than the amount of agreement that is expected due to chance. In essence, Scott’s pi compares coding results to what would happen if two people coded the data at random. ...