Mean Square Error (MSE)

The problem with speaking about the average error of a given statistical model is that it is difficult to determine how much of the error is due to the model and how much is due to randomness. The mean square error (MSE) provides a statistic that allows for researchers to make such claims. MSE simply refers to the mean of the squared difference between the [Page 456]predicted parameter and the observed parameter. Formally, this can be denned as

In Equation (1), E represents the expected value of the squared difference between an estimate of an unknown parameter (θ∗) and the actual observed value (θ) of the parameter. In this instance, the expected value of the MSE simply refers to the average error one would expect given the parameter estimate. MSE is often categorized as a “loss function,” meaning that it represents how wrong the estimated parameter actually is, allowing one to then calculate the parameter's impact on the rest of the model. However, unlike other loss functions, MSE is convex everywhere.

Substantively, the MSE value can be interpreted in many different ways. Statistically, the goal of any model should be to reduce the MSE, since a smaller MSE implies that there is relatively little difference between the estimated and observed parameters. Generally speaking, a well-fitted model should have a relatively low MSE value. The ideal form has an MSE of zero, since it indicates that there is no difference between the estimated and observed parameters. This means that a relatively low MSE should be somewhat close to zero. This interpretation can also be used to compare competing models, using the MSE value as a rubric for deciding which model is best. The model that has the lowest MSE should be considered to be the best, since it provides the best fit and provides the least biased estimate. However, MSE should be used in conjunction with other statistics, such as Adjusted-R2, in order to ensure that the researcher is choosing the best possible model.

MSE is also valuable when it is thought of as a composite of the variance of the estimated parameter and some unknown random bias. Specifically, this can be denned as

Using Equation (2), we can say that an unbiased parameter estimate should have an MSE equal to the variance of the estimated parameter, whereas a biased parameter estimate will have a residual value that represents the squared parameter bias. This is helpful in terms of model building since it allows the researcher to speak in terms of the variance explained by the model and the variance left to random error. A model that has a nonzero bias term can be somewhat problematic since the MSE value serves as the basis for the coefficient standard error, which is then compared to the coefficient magnitude to create the t statistic. A biased MSE can affect these estimates in many ways.

A positive bias term implies that the estimated value is higher than the true value ultimately drawing the t statistic closer to zero, resulting in an increase in Type II error. A negative bias term implies that the estimated value is lower than the true value, which pushes the t statistic away from zero, resulting in an increase in Type I error. Additionally, a relatively low MSE value does not necessarily imply that the parameter estimate is unbiased, since a relatively high bias term can be compensated for by a minimal variance in the estimated parameter. All of these things should be kept in mind when using the MSE value for variable selection and model comparison.

...