Type I and Type II Errors

Type I Error

The probability of a Type I error, also known as alpha (α), is the probability of concluding that a difference exists between groups when in truth it does not. Another way to state this is that alpha represents the probability of rejecting the null hypothesis when it should have been accepted. Alpha is commonly referred to as the significance level of a test or, in other words, the level at or below which the null hypothesis is rejected. It is often set at 0.05 which, although arbitrary, has a long history that originated with R. A. Fisher in the 1920s. The alpha level is used as a guideline to make decisions about the p value that is calculated from the data during statistical analysis: Most typically, if the p value is at or below the alpha level, the results of the analysis are considered significantly different from what would have been expected by chance. The p value is also commonly referred to as the significance level and is often considered analogous to the alpha level, but this is a misuse of the terms. There is an important difference between alpha and p value: Alpha is set by the researcher at a certain level before data are collected or analyzed, while the p value is specific to the results of a particular data analysis. For instance, a researcher might state that he or she [Page 1053]would use an alpha level of 0.05 for a particular analysis. This means two things: First, that he or she accepts the fact that if his or her analysis was repeated an infinite number of times with samples of equal size drawn from the same population, 5% of the time the analysis will return significant results when it should not (a Type I error) and that results with p values of 0.05 or less will be considered significant—that is, not due to chance. The p value calculated for a particular experiment can be any number between 0 and 1: In this example, a p value of 0.02 would be considered significant while a p value of 0.8 would not be.

As an example of a Type I error, consider the case of two normally distributed populations whose true means are equal. If an infinite number of samples are drawn from those populations, the means of the samples will not always be equal, and sometimes will be quite discrepant. Because in most cases we do not know the true population means, we use statistics to estimate how likely the differences in the means found in our samples are, if the population means were truly the same. In doing this, we accept that in some percentage of the cases, we will make the wrong decision, and conclude that the population means are different when they are truly the same: The probability of making this incorrect decision is Type I error or alpha.

...