Stepwise Regression

Stepwise, also called stagewise, methods in fitting regression models have been extensively studied and applied in the past 50 years, and they still remain an active area of research. In many study designs, one has a large number K of input variables and the number n of input-output observations (xi1,$\dots$, xiK, yi), 1 ≤ i ≤ n, is often of the same or smaller order of magnitude than K. Examples include gene expression studies, where the number K of genomic locations is typically larger than the number n of subjects, and signal or image reconstruction, where the number K of basis functions to be considered exceeds the number n of measurements or pixels. Stepwise methods are perhaps the only computationally feasible ways to tackle these problems, and certain versions of these methods have recently been shown to have many desirable statistical properties as well.

Stepwise regression basically carries out two tasks sequentially to fit a regression model

$$y_i={\beta }^T{x}_i+{\epsilon }_i,1\le i\le n,$$

where $\beta ={({\beta }_1,\text{}\dots ,{\beta }_K)}^T$ is a vector of regression parameters, $x_i={(x_{i1},\text{}\dots ,x_{iK})}^T$ is a vector of regressors (input variables), $\epsilon$i represents unobservable noise, and yi is the observed output. The first task is to choose regressors sequentially and the second task is to refit the regression model by least squares after a regressor has been added to the model. For notational simplicity, assume that the yi and xij in Equation 1 have been centered at their sample means so that Equation 1 does not have an intercept term.

To begin, stepwise regression chooses the regressor that is most correlated to the output variable (i.e., such that the [sample] correlation coefficient between yi and xij is the largest among the K regressors). One then performs least squares regression of yi on the selected regressor xij, yielding the least squares fit ${\widehat{y}}_i={\widehat{b}}_j{x}_{ij}$ and the residuals $y_i-{\widehat{y}}_i$. A variable selection criterion is then applied to determine whether the chosen regressor should indeed be included. If the criterion accepts the chosen regressor, then the researcher repeats the stepwise procedure to the remaining regressors but with $y_i-{\widehat{y}}_i$ in place of yi. More generally, after the regressors labeled $j_1,\dots ,j_k$ have been included in the model and the residuals $e_i=y_i-({\widehat{b}}_{j1}{x}_{i,\text{}j1}+\cdots +{\widehat{b}}_{jk}{x}_{i,\text{}jk})$ have been computed, the researcher chooses $\tilde{j}\notin \left\{j_1,\dots ,j_k\right\}$ such that the correlation coefficient between ei and $x_{i,\tilde{j}}$ is the largest among the remaining $K-k$ input variables, and it performs least squares regression of ei on $x_{i,\tilde{j}}$, yielding a [Page 1622]new set of residuals ei, which are used in the criterion to determine whether the regressor labeled $\tilde{j}$ should be included. If the criterion rejects the regressor, then it is not included in the model, and the stepwise regression procedure terminates with the set of input variables $x_{i,\text{}j1},\dots ,x_{i,jk}$.

A traditional variable selection criterion is based on the F test of H0 : $\beta$j = 0 in the regression model $y_i={\beta }_{j1}{x}_{i,\text{}j1}+\cdots +{\beta }_{jk}{x}_{i,jk}+{\beta }_{\tilde{j}}{x}_{i,\tilde{j}}+{\epsilon }_i$ to determine whether the regressor labeled $\tilde{j}$ should be added to the model that already contains the regressors labeled $j_1,\dots ,j_k$. If the F test rejects H0 at significance level $\alpha$, which is often chosen to be 5%, then the regressor labeled j is included in the model. Otherwise, ${\beta }_j$ is deemed to be not significantly different from 0, and therefore the corresponding regressor is excluded from the model. Note that because this test-based procedure carries out a sequence of F tests, the overall significance level can differ substantially from α. Such an F test of whether a particular regressor in a larger set of input variables has regression coefficient 0 is called a partial F test, and the corresponding test statistic is called a partial F statistic.

...