Significance Level

The significance level (also called Type I error rate or the level of statistical significance) refers to the probability of rejecting a null hypothesis that is in fact true. This quantity ranges from zero (0.0) to one (1.0) and is typically denoted by the Greek letter alpha (a). The significance level is sometimes referred to as the probability of obtaining a result by chance alone. As this quantity represents an “error rate,” lower values are generally preferred. In the literature, nominal values of a generally range from 0.05 to 0.10. The significance level is also referred to as the “size of the test” in that the magnitude of the significance level determines the end points of the critical or rejection region for hypothesis tests. As such, in hypothesis testing, the p-value is often compared to the significance level in order to determine if a test result is “statistically significant.” As a general rule, if the p-value is no larger than the significance level, the null hypothesis is rejected and the result is deemed statistically significant, thus supporting the alternative hypothesis.

The level of significance can refer to the Type I error for a single hypothesis test or for a family of simultaneous tests. In the latter case, the “experiment-wise” or “family-wise” significance level refers to the probability of making at least one Type I error over the collection of hypothesis tests that are contained in the family. So for example, a survey may contain questions to solicit data to be used to compare the average expenditures, household ownership percentage, and education levels across two possible geographic sectors of a particular county. Because there are three main variables of interest that are to be compared across the two geographical regions, the family-wise level of significance will refer to the probability of making a Type I error rate on at least one of the three hypothesis tests that are performed for this family of tests.

In another example of multiple comparisons, comparisons of the average length of unemployment were made across four racial post-strata after an omnibus analysis of variance (ANOVA) revealed that the average unemployment periods are not equal across the four race groups. One possible set of post-hoc multiple comparisons consists of all pairwise tests for differences in the average unemployment period for two race groups at a time (i.e. six pairwise tests).

A Bonferroni adjustment or other multiple comparisons adjustment is typically made to the overall nominal Type I error rate to ensure the proper significance level is achieved for the family of tests. For example, to ensure that the overall significance level for a family of three hypothesis tests, the nominal significance level, α would be divided by 3, and the adjusted significance level, α/3, would be used as the Type I error rate for each of the three hypothesis tests in the family. In the second scenario, six comparisons would be made, so the Bonferroni adjustment for multiple comparisons equates to using α/6 as the Type I error rate for each of the pairwise comparisons.

...