Principal Components Analysis

Theory

Suppose that we have the scores of I individuals on J variables and that the relationships between the variables are such that no variable can be perfectly predicted by all the remaining variables. Then these variables form the axes of a J-dimensional space, and the scores of the individuals on these J variables can be portrayed in this J-dimensional space. However, looking at a high-dimensional space is not easy; moreover, most of the variability of the high-dimensional arrangement of the individuals can often be displayed in a low-dimensional space without much loss in variability. As an example, we see in Figure 1 that the two-dimensional ellipse A of scores of Sample A can be reasonably well represented in one dimension by the first principal component, and one only needs to interpret the variability along this single dimension. However, for the scores of Sample B, the one-dimensional representation is much worse (i.e., the variance accounted for by the first principal component is much lower in Case B than in Case A, and interpreting a single dimension might not suffice in Case B).

The coordinate axes of the low-dimensional dimensional space are commonly called components. If the components are such that they successively account for most of the variability in the data, they are called principal components. The coordinates of the individuals on the components are called component scores. To interpret components, the coordinates for the variables [Page 856]on these components need to be derived as well, and the common approach to do this is via EIGENVALUEEIGENVECTOR techniques. If both the variables and the components are standardized, the variable coordinates are the correlations between variables and components. By inspecting these correlations, commonly known as component loadings, one may assess the extent to which the components measure the same quantities as (groups of) variables. In particular, when a group of variables has high correlations with a component, the component has something in common with all of them, and on the basis of the substantive content of the variables, one may try to ascertain what the common element between the variables may be and hypothesize that the component is measuring this common element.

...