Cross-Validation

K-Fold Cross-Validation

In many applications, the amount of available data is not large enough for a simple cross-validation. Instead, K-fold cross-validation is commonly used to extract more information from the data. Unlike the simple training-and-testing division, the available data are randomly divided into K roughly equal parts. Each part is chosen in turn for testing purposes, and each time the remaining $(K-1)$ parts are used for training purposes. The prediction errors from all the K validations are collected, and the sum is used for cross-validation purposes.

In order to formulate the K-fold cross-validation using common notations, suppose the available data set consists of N observations or data points. The ith observation includes predictor(s) $x_i$ in scalar (or vector) form and response $y_i$, also known as input and output, respectively. Suppose a random partition of the data divides the original index set $\left\{1,2,\dots ,N\right\}$ into K subsets $I(1),I(2),\dots ,I(K)$ with roughly equal sizes. For the kth subset, let ${\widehat{f}}^{-k}(\cdot )$ be the fitted prediction function based on the rest of the data after removing the kth subset. Then the K-fold cross-validation targets the average prediction error defined as

$$\text{CV}=\frac{1}{N}{\displaystyle \sum _{k=1}^K{\displaystyle \sum _{i\in I(k)}L\left(y_i,{\widehat{f}}^{-k}({\text{x}}_i)\right)}}.$$

In the aforementioned expression, CV is average prediction error, $L(·,·)$ is a predetermined function, known as the loss function, which measures the difference between the observed response $y_i$ and the predicted value ${\widehat{f}}^{-k}({\text{x}}_i)$. Commonly used loss functions $L(y,\widehat{f})$ include the squared loss function ${\left(y-\widehat{f}\right)}^2$; the absolute loss function $\left|y-\widehat{f}\right|$; the 0−1 loss function, which is $0\text{ if }y=\widehat{f}\text{ and }1\text{ otherwise}$; and the cross-entropy loss function $-2\log \widehat{P}(Y=y_i|{\text{x}}_i)$, when a predicted probability is available.

The result CV of K-fold cross-validation depends on the value of K used. Theoretically, K can be any integer between 2 and the data size N. Commonly used values of K include 5 and 10. Generally speaking, when K is small, CV tends to overestimate the prediction error because each training part is only a fraction of the full data set. As K gets closer to N, the expected bias tends to be smaller, while the variance of CV tends to be larger because the training sets become more and more similar to each other. In the meantime, the cross-validation procedure involves more computation because the target model needs to be fitted for K times. As a compromise, 5-fold or 10-fold cross-validation would be suggested.

...