Factor Analysis and Principal Components Analysis

Principal Components

PCA can be used to reduce the dimensionality of data in the sense of transforming an original set of variables to a smaller number of transformed ones. Such a purpose is desirable as it allows for the parsimonious explanation of the systematic variation of data with as few variables as possible. Obtaining parsimonious representations of data is especially useful when confronted with large numbers of variables, such as those found in survey data or genetics data. Socioeconomic variables have been combined into a smaller number through PCA as well. Furthermore, in regression analyses, multicollinearity can be a serious concern when there are a large number of variables to model. Reducing the number of variables used in an analysis or transforming the original variables to make them uncorrelated, as PCA does, can alleviate this problem.

PCA involves rotating multivariate data, which involves transforming the original variables into a new set of variables that are linear combinations of the original variables. This rotation process yields a new set of variables with desirable properties.

Let X1, …, X (the Xs) denote the original variables. For instance, the Xs could be clinical variables, such as X1being weight measurements, X2 being heights, X3 being systolic blood pressure, and so on. Each Xi, i = 1, …, p, is a vector with n elements, representing, for instance, n observations of the variable Xi from n subjects. A linear combination of the Xs would take the form a1X1 +…+ apX, for some constant weights a1, …, ap. Loosely speaking, one object of PCA is to find uncorrelated linear combinations of the Xs that maximize the variance, a measure of the variability in data. Weights for the linear combinations being considered are restricted so that the sum of their squared values is 1. This restricts possible solutions under consideration to be derivable from rotations. Based on elegant theories from linear algebra, a sketch of how they are derived is given below (for more details, see Tatsuoka, 1988).

Given variables X1, …, X, one can construct a p × p matrix A that is composed of sample covariances A, with the i,jth entry in A corresponding to the sample covariance between Xi and X. Covariances measure the degree to which two variables vary together, or are correlated. We can solve what is known as the characteristic equation for the matrix A and generate p nonnegative roots (although it is possible that some roots are equal, [Page 504]or even zero, depending on the rank of A). This equation is derived based on the objective of finding linear combinations of the Xs that maximize variance. These roots are known as the eigenvalues of the matrix A. Furthermore, given these eigenvalues, corresponding vectors, called eigenvectors, can be derived where the elements in the eigenvectors give the weights for the desired linear combinations. Moreover, the eigenvalues are equal to the corresponding variance of the linear combination of the Xs, with weights corresponding to the eigenvector. Hence, the eigenvalues and eigenvectors that are generated provide the essential practical information in attaining the objectives of PCA.

...