Accuracy Assessment

Keywords

cross validation, design-based inference, error matrix, land-cover change, probability sampling, reference data error.

Introduction

Remote sensing can be used to generate products representing a variety of features of the Earth's environment. These products may consist of categorical outputs such as maps of land-cover or continuous outputs such as maps of surface temperature or leaf area index. Assessing the accuracy of these products is critical to understanding their potential utility and the possible impact of error in the product on the intended application. Accuracy assessment is an important element of ‘validation’, where validation is defined as ‘the process of assessing by independent means the quality of the data products derived from the system outputs’ (Justice et al. 2000, p. 3383).

Accuracy is defined as the degree to which the map corresponds to what is on the ground. Therefore, the fundamental basis of accuracy assessment is a comparison of the derived product to ground condition. Typically, it is not possible to ascertain perfectly the ground condition, so in reality we assess ‘agreement’ with the best available information on ground condition rather than ‘accuracy’ based on the true ground condition. In recognition of the limitations of the data on ground condition it is also preferable to avoid the expression ‘ground truth’ and instead refer to ground or reference data.

An accuracy assessment consists of three components, a response design, sampling design, and analysis (Stehman and Czaplewski 1998). The response design is the protocol for collecting information to determine the ground condition associated with a point location, a pixel, an object, or an areal unit, and translating that information into a label or quantity against which the map label or quantity is compared. Because it is usually impractical to determine the ground condition for the entire area mapped, sampling is a fundamental component of accuracy assessment. The sampling design specifies the locations at which the ground data will be collected. The analysis protocol specifies the measures used to describe accuracy and how to estimate these measures from the sample.

One of the most common applications of accuracy assessment related to terrestrial remote sensing is evaluating the accuracy of categorical products such as land-cover produced by a supervised classification analysis. Two types of classification are commonly employed. In a hard or crisp classification, each pixel is assigned to one and only one class, whereas for a soft classification, each pixel is assigned a grade of membership for each class. Applications in which both the map and ground data classifications are hard are the most prevalent and the focus of attention in this chapter. But recognition of the ‘mixed pixel’ problem (i.e., pixels representing an area of more than one class) has led to greater use of methods involving soft classification.

The accuracy of categorical products is typically evaluated using a ‘site-specific’ approach in which the map and ground data are compared on a location-by-location basis. In contrast, ‘nonsite-specific’ accuracy assessment compares the proportion of area classified or mapped for [Page 298]each class to the corresponding true proportion of area of each class, and thus evaluates an aggregate feature of the map. With a site-specific approach, each location is associated with two labels, the class predicted by the classification that is shown on the map and the class observed as the ground condition. Cross-tabulating the set of class labels provides a simple summary of the quality of a classification. The derived confusion or error matrix highlights those cases for which the labels agree and the classification is accurate (main diagonal of Table 21.1), and, where there is error (off-diagonal elements of Table 21.1), the nature of the error is indicated (e.g., omission and commission error). The matrix may be used to derive estimates of accuracy on a per-class as well as overall basis. Per-class accuracy is defined from two perspectives, producer's accuracy (the complement of omission error) and user's accuracy (the complement of commission error).

...