Interrater Reliability

The use of raters or observers as a method of measurement is prevalent in various disciplines and professions (e.g., psychology, education, anthropology, and marketing). For example, in psychotherapy research raters might categorize verbal (e.g., paraphrase) and/or nonverbal (e.g., a head nod) behavior in a counseling session. In education, three different raters might need to score an essay response for advanced placement tests. This type of reliability is also present in other facets of modern society. For example, medical diagnoses often require a second or even third opinion from physicians. Competitions, such as Olympic figure skating, award medals based on quantitative ratings provided by a panel of judges.

Those data recorded on a rating scale are based on the subjective judgment of the rater. Thus, the generality ...