Brier Scores

General

Consider an event that can only occur in one of r distinct categories. For example, a medical decision maker might assign probabilities to each category, where the probabilities should sum up to 1. Let πj∊ [0,1] denote the prediction for the probability that the event occurs in category j, for j = 1,…, r, and let Yj denote the random outcome, where Yj = 1 if the event occurs in category j and Yj = 0 if it does not. The Brier score is a loss function that has been proposed as a measure to quantify the loss incurred if π is predicted, and Y is the outcome. It is the squared difference(π – Y)2. In a sample of size n where πij and yij are the ith prediction and the ith actually observed outcome for category j, respectively, the empirical Brier score is given by (1/n)ΣiΣj(πij—yij)2. For example, when the events “relapse” versus “no relapse” are of interest, and there are two patients, the first with a relapse and the second without, a naive predicted probability of .5 for both patients results in a Brier score of 1/2 × (((.5 − 0)2 + (.5 − 1)2) + ((.5 − 1)2 + (.5 − 0)2)) = .5. If, however, for the patient with relapse, the predicted probability of relapse is .6, and for the patient without relapse the predicted probability of relapse is .3, then the Brier score reduces to 1/2 × (((.4 − 0)2+ (.6 − 1)2) + ((.7 − 1)2 + (.3 − 0)2)) = .25.

Brier Scores with Dichotomous Data

In a setting with dichotomous data, that is, with only r = 2 categories (e.g., when one predicts whether a patient will survive for a certain period of time), it is common to consider only one of the categories for calculation (dropping the subscript j). While in the original formulation, the Brier score takes values in the range from 0 to 2 when there are only 2 categories, the modified version ranges from 0 to 1; that is, the resulting value is only half of the original Brier score.

...