Skip to main content icon/video/no-internet

Reliability can be defined as the extent to which scores of a measure are free from the effect of measurement error. Measurement error is reflected in random deviations of the scores observed on a measure from respondents' true scores, which are the expected values of respondents' scores if they completed the measure an infinite number of times. Mathematically, reliability is quantified as the ratio of true score variance to observed score variance or, equivalently, the square of the correlation between true scores and observed scores. Based on these indexes, reliability can range from zero (no true score variance) to one (no measurement error).

Reliability is important for both practical and theoretical purposes. Practically, it enables estimation of the standard error of measurement, an index of accuracy of a person's test score. Theoretically, reliability contributes to theory development by allowing researchers to correct for the biasing effect of measurement error on observed correlations between measures of psychological constructs and by providing researchers with an assessment of whether their measurement process needs to be improved (e.g., if reliability is low).

Sources of Measurement Error

Multiple sources of measurement error can influence a person's observed score. The following sources are common in psychological measures.

Random Response Error

Random response error is caused by momentary variations in attention, mental efficiency, or distractions within a given occasion. It is specific to a moment when a person responds to an item on a measure. For example, a person might provide different answers to the same item appearing in different places on a measure.

Transient Error

Whereas random response error occurs within an occasion, transient error occurs across occasions. Transient errors are produced by temporal variations in respondents' mood and feelings across occasions. For example, any given respondent might score differently on a measure administered on two occasions. Theoretically, such temporal differences are random, and thus not part of a person's true score, because they do not correlate with scores from the measure completed on other occasions (i.e., they are occasion specific).

Specific Factor Error

Specific factor error reflects idiosyncratic responses to some element of the measurement situation. For example, when responding to test items, respondents might interpret item wording differently. Theoretically, specific factors are not part of a person's true score because they do not correlate with scores on other elements (e.g., items) of the measure.

Rater Error

Rater error arises only when a person's observed score (rating) is obtained from another person or set of persons (raters). Rater error arises from the rater's idiosyncratic perceptions of a ratee's standing on the construct of interest. Theoretically, idiosyncratic rater factors are not part of a person's true score because they do not correlate with ratings provided by other raters (i.e., they are rater specific).

Types of Reliability Coefficients

Reliability is indexed with a reliability coefficient. There are several types of reliability coefficients, and they differ with regard to the sources of observed score variance that they treat as true score and error variance. Sources of variance that are treated as error variance in one type of coefficient may be treated as true score variance in other types.

...

  • Loading...
locked icon

Sign in to access this content

Get a 30 day FREE TRIAL

  • Watch videos from a variety of sources bringing classroom topics to life
  • Read modern, diverse business cases
  • Explore hundreds of books and reference titles

Sage Recommends

We found other relevant content for you on other Sage platforms.

Loading