Rating

Ratings are summaries of behaviors or attitudes that have been organized according to a set of rules. Although the general use of the term in research can refer to written data, such as an anecdotal report, a rating usually refers to a number assigned according to a defined scale.

Ratings can be quantified, as numbers or scores chosen to identify the position on a scale, or they can be graphic indicators. Likert-type scales, for example, provide a balanced list of alternatives from which raters may choose to indicate a response to a statement or question. The response chosen is attached to a quantity that becomes a score for use in data analysis. A rating item designed in this way might appear as shown below:

This type of rating is so common that it appears intuitive, but it is not without its disadvantages. Concerns about of this method can arise when the data are analyzed statistically. Most researchers are comfortable treating these responses, and especially summed or averaged total scores across many such ratings, as interval-level measurement and apply parametric statistical methods, which assume that high level of measurement. Some critics point out that the psychological processes that lead to a rater's choosing a particular rating may not reflect an underlying interval-level scale (e.g., is the meaningful “distance” between strongly agree and agree ratings equal to the distance between disagree and strongly disagree?). So the use of parametric statistics may be inappropriate. The consensus of studies over the past several decades, however, suggests that the use of parametric statistical methods to analyze ratings of this type produces fairly accurate results, and the concern may be primarily academic.

Examples of graphic indicators as ratings include visual analog scales, which can be used to indicate with a high degree of precision a respondent's subjective impression. This method allows for a greater variability in responses and, theoretically, allows for the score to more closely estimate a true or accurate response. Pain scales, for example, ask patients to mark an “x” somewhere along a line to indicate the level of pain they are currently experiencing: No pain——————x————Most pain

The score for a graphic rating scale is determined by literally measuring the location of the mark along the line, in millimeters, for example, and that length becomes the rating. Beyond the potential for greater precision, an advantage of this rating method is that the measurement approach is more clearly interval- or ratio-level, and parametric statistical methods seem a clearly appropriate method of analysis. Concerns for this method center on the absence of many anchors, verbal description of scale points, and logistical measurement difficulties when forms are reproduced at different sizes, such as in different online contexts.

A third type of rating, the checklist, is sometimes quantified and sometimes represented as a set of independent observations. Checklists are lists of characteristics or components that may be present or absent in a person, object, or environment. A scale for depression, for example, or alcoholism, may ask people to place a checkmark next to, or circle, each characteristic that is true of them. The number of checks may be summed, and if a cut score, or critical total, is reached, then the raters may be categorized as, for example, being depressed or having a drinking problem.

...