Maximum Likelihood

Origins

R. A. Fisher invented the method of maximum likelihood in a series of papers published early in the 20th century. Fisher's work on maximum [Page 1510]likelihood began with his derivation of the principle of “absolute criterion” in a paper he published as a third-year undergraduate. While this paper contains the origins of maximum likelihood estimation (MLE), there is little in the paper that many readers would recognize as MLE. In later papers, he developed the concept of likelihood as distinct from probability. Then, in 1922, Fisher united several earlier streams of his research and was the first to use the term maximum likelihood for a class of estimators as an alternative to Bayesian or method of moments estimators. In the same paper, Fisher proposes that maximum likelihood estimators have properties of efficiency, sufficiency, and consistency. Later work by other statisticians would establish the properties of MLE more rigorously.

A simple example is helpful for understanding the principles of MLE. Let us say we wish to estimate the sample proportion for a set of data. Assume we have a random sample of data y1, …, yn with n observations randomly drawn from a binomial distribution with common parameter p, where 0 < <p < 1 and each <y is either 1 for success or 0 for failure. For these n independent and identically distributed variables y1, …, yn, the density of each observation is

We next write the likelihood function, which is the density evaluated at the data as a function of the parameter p. However, because the binomial coefficient does not depend on the parameter of interest, p, we can omit it from our derivation of the maximum likelihood estimator. The likelihood function is the product of the individual densities for each observed data

...