Weighted Least Squares

Motivation

Standard regression modeling assumes that the variance of the residuals is constant across the levels of the explanatory variables. Often in political science this is violated, causing potential problems of fit and interpretation. George Downs and David Rocke (1979) give the well-known examples of increasing spread around consumption as income increases, increasing spread in the proportion of felons incarcerated in states as legislative interest increases, and decreasing spread in that proportion as the average education level in the state decreases. All of these are examples where one aspect of the relationship between the explanatory variable and the outcome variable, outcome uncertainty, changes as the levels of the right-hand side are altered. WLS addresses this problem by modeling this heteroscedasticity rather than ignoring it. Thus, there is a trade-off between increased model complexity and increased interpretational complexity.

Theory

The ordinary linear model has the form y = Xβ + ε, where y is an n × 1 outcome vector with continuous measure, X is an n × k nonsingular matrix with explanatory variables down the columns and a leading column of ones, β is a k × 1 parameter vector to be estimated, and ε is an n × 1 error vector with assumed mean zero. The ordinary least squares (OLS) estimator of β is arrived at by minimizing the squared error terms and is formed by (X'X)–1X'y. In the presence of heteroscedasticity, the OLS estimator of β is not the best linear [Page 2740]unbiased estimator (BLUE). The term best here means that the estimator achieves the minimum possible variance.

WLS allows one to reformulate the model and generate estimates that are in principle the BLUE. The introduction of a weight matrix Ω into the calculation of removes the heteroscedasticity from the model. The Ω matrix is created by taking the error variance of the ith case (estimated or known), vi, and assigning it to the ith diagonal, Ωii = 1/vi = ωi, leaving the off-the-diagonal elements as zero. In this way, large error variances are reduced by premultiplying the model terms by this reciprocal. We can premultiply each term in the standard linear model setup by the square root of the Ω matrix (i.e., by the standard deviation). This “square root” is actually produced from a Cholesky factorization: If A is a positive definite symmetric (A' = A) matrix, then there must exist a matrix G such that A = GG'. A matrix A is positive definite if for any nonzero p × 1 vector x, x'Ax > 0. In our case, this decomposition is greatly simplified because the Ω matrix has only diagonal values (all off-the-diagonal values equal zero). Therefore, the Cholesky factorization is produced simply from the square root of these diagonal values. Premultiplying

...