Gauss-Markov Theorem

Origins

In 1821, Carl Friedrich Gauss proved that the least squares method produced unbiased estimates that have the smallest variance, a result that has become the cornerstone of regression analysis. In his 1900 textbook on probability, Andrei Markov essentially rediscovered Gauss's theorem. By the 1930s, however, the result was commonly referred to as the Markov theorem rather than the Gauss theorem. Perhaps because of awareness of this mis-attribution to Markov, many statisticians today avoid using the Gauss or Markov label altogether, referring instead to the equivalence of ordinary least squares (OLS) estimator and best linear unbiased (BLU) estimator. Most econometricians, however, refer to the result instead by the compromise label used here, the Gauss-Markov theorem.

Statistical Estimation and Regression

The goal of statistical estimation is to provide accurate guesses about parameters (statistical summaries) of a population from a subset or sample of the population. In regression estimation, the main population parameters of interest measure how changes in one (independent) variable influence the value of another (dependent) variable. Because statistical estimation involves estimating unknown population parameters from known sample data, there are actually two equations involved in any simple regression estimation: a population regression function, which is unknown and being estimated,

and a sample regression function, which serves as the estimator and is calculated from the available data in the sample,

where

y = dependent variable

x = independent variable

α = y-intercept of population regression function

β = slope of population regression function

μ = error, disturbance of population regression function

Because statistical estimation involves the calculation of population parameters from a finite sample of data, there is always some uncertainty about how close statistical estimates are to actual population parameters. To sort out the many possible ways of estimating a population parameter from a sample of data, various properties have been proposed for estimators. The “ideal estimator” always takes the exact value of the population parameter it is estimating. This ideal is unachievable from a finite sample of the population, and estimation instead involves a trade-off between different forms of accuracy, such as unbiasedness and minimum variance. The best linear unbiased estimator, which is discussed next, represents such a trade-off.

...