Holm's Sequential Bonferroni Procedure

The more statistical tests one performs the more likely one is to reject the null hypothesis when it is [Page 574]true (i.e., a false alarm, also called a Type 1 error). This is a consequence of the logic of hypothesis testing: The null hypothesis for rare events is rejected in this entry, and the larger the number of tests, the easier it is to find rare events that are false alarms. This problem is called the inflation of the alpha level. To be protected from it, one strategy is to correct the alpha level when performing multiple tests. Making the alpha level more stringent (i.e., smaller) will create less errors, but it might also make it harder to detect real effects. The most well-known correction is called the Bonferroni correction; it consists in multiplying each probability by the total number of tests performed. A more powerful (i.e., more likely to detect an effect exists) sequential version was proposed by Sture Holm in 1979. In Holm's sequential version, the tests need first to be performed in order to obtain their p values. The tests are then ordered from the one with the smallest p value to the one with the largest p value. The test with the lowest probability is tested first with a Bonferroni correction involving all tests. The second test is tested with a Bonferroni correction involving one less test and so on for the remaining tests. Holm's approach is more powerful than the Bonferroni approach, but it still keeps under control the inflation of the Type 1 error.

The Different Meanings of Alpha

When a researcher performs more than one statistical test, he or she needs to distinguish between two interpretations of the α level, which represents the probability of a Type 1 error. The first interpretation evaluates the probability of a Type 1 error for the whole set of tests, whereas the second evaluates the probability for only one test at a time.

Probability in the Family

A family of tests is the technical term for a series of tests performed on a set of data. This section shows how to compute the probability of rejecting the null hypothesis at least once in a family of tests when the null hypothesis is true.

For convenience, suppose that the significance level is set at α = .05. For each test the probability of making a Type I error is equal to α = .05. The events “making a Type I error” and “not making a Type I error” are complementary events (they cannot occur simultaneously). Therefore, the probability of not making a Type I error on one trial is equal to

Recall that when two events are independent, the probability of observing these two events together is the product of their probabilities. Thus, if the tests are independent, the probability of not making a Type I error on the first and the second tests is

With three tests, the probability of not making a Type I error on all tests is

For a family of C tests, the probability of not making a Type I error for the whole family

...