Residuals

Regression Analysis

The concept of residual in regression analysis is of utmost importance, particularly when one is [Page 1268]applying the regression method of ordinary least squares. In ordinary least squares regression analysis, the goal of which is to fit a line or (hyper)plane to observed data when one is using more than one explanatory variable, residuals depict the vertical distances between observed values and the corresponding fitted values. Conceptually, the residual is the part of the dependent variable (Y) that is not linearly related to the explanatory variables (X1, X2, …, Xp). In other words, it is the part of Y that cannot be explained by the estimated regression model.

In a regression model defined as

the ith fitted value, the value lying on the hyper-plane, can be calculated as

where p refers to the number of explanatory variables and n, the number of observations.

Analogously, the vertical distance between observed and fitted values for the ith observation, the ith residual (ei), is defined as

which is graphically shown for the simple regression model in Figure 1.

Figure 1 Residual (ei) in Ordinary Least Square Regression Analysis

Standard Regression Assumptions

The assumptions about the error terms for the ordinary least square estimation are that the error terms are independently and identically distributed normal random variables, each with a mean of zero and a common variance, often expressed as

where σ2In denotes a (n-n) matrix with the individual error variances on the main diagonal being equal to σ2 and all the covariances on the off-diagonal being zero. For instance, for n = 6, the matrix σ2In is

Because of these error term assumptions, a standard practice in applied regression analysis is to include a series of tests and/or residual plots to check whether the residuals violate some of these assumptions. The literature is rich on statistical tests based on the calculated residuals, which include the Durbin-Watson test for autocorrelation (i.e., the errors are independent of each other) and the Breusch-Pagan test for heteroscedasticity (i.e., constant variance of the errors).

In addition to all these statistical tests, residual plots are an excellent and necessary tool to check for potential violations of the standard regression assumptions. As explained below, however, for better interpretation most residual plots do not use the original estimated residuals but rather use transformed residuals, such as standardized or studentized residuals. For instance, the normal probability plot of studentized residuals is suitable to check for the normality assumption, while the scatterplot of studentized residuals versus the explanatory variables is usable to check for the linearity and the constant variance assumptions. Further, the index plot of studentized residuals will point toward violations of the independence of errors, such as autocorrelation or spatial dependence of the error terms.

...