Weighted Least Squares

Weighted least squares (WLS) is an extension of ordinary least squares (ols) regression that weights each observation according to some criterion, such as differences in the accuracy of estimates or error variances. In regular OLS, slope estimates are found by minimizing the sum of squares, ∑E2i, where E represents the residuals from the regression. That is, we minimize the differences in the square of the actual values from the predicted values from the regression equation. WLS extends this model by minimizing the sum of the weighted least squares ∑w2iE2i, where w represents a weight applied to each observation. Estimates from a WLS regression are interpreted in exactly the same manner as estimates from OLS. WLS is also efficient and shares with OLS the standard procedures for statistical inference (e.g., confidence intervals and hypothesis tests).

WLS is most frequently used in the social sciences when heteroskedasticity (nonconstant error variance) is present in a regression analysis. The two problems associated with heteroskedasticity—inefficient estimates and biased standard errors—can be limited using weighted least squares. WLS can be used when the functional form of the relationship is linear or nonlinear.

Before WLS can be implemented to correct for heteroskedasticity, however, it must be known that the error variance is systematically related to one of the independent variables. Because this is not known before fitting a regression model, weights are typically found by examining the residuals from a preliminary OLS model. A common pattern of heteroskedasticity is the increase in error variance as the value of an independent variable, X, increases. In such cases, the variance of the errors is proportional to X, and thus, 1/X is usually an effective weight for computing the weighted least squares estimates. A plot of the log of the Studentized residuals against the fitted values can help detect this phenomenon.

The disadvantage of using WLS to correct for heteroskedasticity is that we require knowledge of the relationship between the errors and a particular independent variable. Because this cannot always be determined, OLS with robust standard errors is often used as an alternative. Unlike WLS, the use of robust standard errors does not change the OLS regression coefficients, but rather only increases the size of the standard errors. In other words, the use of OLS and robust standard errors will improve statistical testing, but it will not necessarily provide reliable coefficient estimates. As a result, if the pattern of heteroskedasticity can be determined, WLS is a better choice than OLS with robust standard errors.

When a nonrandom pattern in error variance occurs in a time-series, a related method, generalized least squares, is used in place of weighted least squares. Iterated weighted least squares (also called iteratively reweighted least squares) can also be used to fit generalized linear models. Finally, weighted least squares is also used in local regression to find local estimates and in robust regression to lessen the impact of influential cases on the regression estimates.