Weighted Least Squares

Least Squares Methods

The OLS method consists of finding the equation (i.e., slope and intercept) of the line that minimizes the sum of the squared vertical distances between each point and the line, hence the name least squares. These distances are called residuals. A picture helps understand these concepts. In 2001, Frank Bengel and colleagues studied the relationship between percent change in ejection fraction from rest to exercise (y variable) and hydroxyephedrine retention (x variable) in heart transplant patients. Each point in Figure 1 represents a patient and the line was computed using the OLS method. As is apparent from the figure, some points are farther from the line than others. The residuals are the distances from the line to each point in the y direction. By convention, points below the line have a negative residual and those above the line, a positive residual. The well-known correlation coefficient is a measure of the strength of the linear association and depends on the magnitude of the residuals. In this case, r = .61, indicating a moderately positive association that is statistically significant (i.e., not due to chance, since p < .001). An equivalent statement is that the slope of the line is significantly different from zero.

Mathematically, the OLS method minimizes the quantity

where ei is the residual of the ith point and n is the number of points. In this sum, each residual has the same weight, and thus, each point (or residual) contributes equally to the sum. This is consistent with the assumption that every point contains the same amount of information, which is one of the key statistical assumptions of SLR. The (theoretical) assumption is that for each x measure, there exists a distribution of y measures with a mean that depends on the x measure and a variance that is the same for all values of x.

In statistical terms, the uniformity of variances is called homoscedasticity. Frequently, it is assumed that the distribution is normal. Figure 2a illustrates [Page 1186]this assumption. The popularity of OLS is due, in large part, to the fact that, under homoscedasticity (together with the independence of the observations), the fitted line has optimal statistical properties (i.e., the slope and intercept have smallest variance).

...