Least Squares, Methods of

The least-squares method (LSM) is widely used to find or estimate the numerical values of the parameters to fit a function to a set of data and to characterize the statistical properties of estimates. It is probably the most popular technique in statistics for several reasons. First, most common estimators can be cast within this framework. For example, the mean of a distribution is the value that minimizes the sum of squared deviations of the scores. Second, using squares makes LSM mathematically very tractable because the Pythagorean theorem indicates that, when the error is independent of an estimated quantity, one can add the squared error and the squared estimated quantity. Third, the mathematical tools and algorithms involved in LSM (derivatives, eigendecomposition, and singular value decomposition) have been well studied for a long time.

LSM is one of the oldest techniques of modern statistics, and even though ancestors of LSM can be traced back to Greek mathematics, the first modern precursor is probably Galileo. The modern approach was first exposed in 1805 by the French mathematician Adrien-Marie Legendre in a now classic memoir, but this method is somewhat older because it turned out that, after the publication of Legendre's memoir, Carl Friedrich Gauss (the famous German mathematician) contested Legendre's priority. Gauss often did not publish ideas when he thought that they could be controversial or not yet ripe, but he would mention his discoveries when others would publish them (the way he did, for example for the discovery of non-Euclidean geometry). And in 1809, Gauss published another memoir in which he mentioned that he had previously discovered LSM and used it as early as 1795 in estimating the orbit of an asteroid. A somewhat bitter anteriority dispute followed (a bit reminiscent of the Leibniz-Newton controversy about the invention of calculus), [Page 706]which however, did not diminish the popularity of this technique.

The use of LSM in a modern statistical framework can be traced to Sir Francis Galton who used it in his work on the heritability of size, which laid down the foundations of correlation and (also gave the name to) regression analysis. The two antagonistic giants of statistics Karl Pearson and Ronald Fisher, who did so much in the early development of statistics, used and developed it in different contexts (factor analysis for Pearson and experimental design for Fisher).

Nowadays, the LSM exists with several variations: Its simpler version is called ordinary least squares (OLS), and a more sophisticated version is called weighted least squares (WLS), which often performs better than OLS because it can modulate the importance of each observation in the final solution. Recent variations of the least square method are alternating least squares (ALS) and partial least squares (PLS).