Skip to main content icon/video/no-internet
icon backReturn to Entries

Bonferroni Test

The more tests we perform on a set of data, the more likely we are to reject the null hypothesis when it is true (a Type I error). This is a consequence of the logic of hypothesis testing: We reject the null hypothesis if we witness a rare event. But the larger the number of tests, the easier it is to find rare events, and therefore, the easier it is to make the mistake of thinking that there is an effect when there is none. This problem is called the inflation of the alpha level. One strategy for preventing it is to correct the alpha level when performing multiple tests. Making the alpha level more stringent (i.e., smaller) will create fewer errors, but it may also make real effects harder to detect.

The Different Meanings of Alpha

Maybe researchers perform more and more statistical tests on one set of data because computers make statistical analyses easy to run. For example, brain imaging researchers will routinely run millions of tests to analyze an experiment. Running so many tests increases the risk of false alarms. To illustrate, imagine the following “pseudoexperiment”:

I toss 20 coins, and I try to force the coins to fall heads up. I know that, from the binomial test, the null hypothesis is rejected at the α= .05 level if the number of heads is greater than 14. I repeat this experiment 10 times.

Suppose that one trial gives the “significant” result of 16 heads versus 4 tails. Did I influence the coins on that occasion? Of course not, because the larger the number of experiments, the greater the probability of encountering a low-probability event (like 16 versus 4). In fact, waiting long enough is a sure way of detecting rare events!

Probability in the Family

A family of tests is the technical term for a series of tests performed on a set of data. In this section, we show how to compute the probability of rejecting the null hypothesis at least once in a family of tests when the null hypothesis is true.

For convenience, suppose that we set the significance level at α = .05. For each test (i.e., one trial in the example of the coins), the probability of making a Type I error is equal to α= .05. The events “making a Type I error” and “not making a Type I error” are complementary events (they cannot occur simultaneously). Therefore the probability of not making a Type I error on one trial is equal to

None

Recall that when two events are independent, the probability of observing these two events together is the product of their probabilities. Thus, if the tests are independent, the probability of not making a Type I error on the first and the second tests is

None

With three tests, we find that the probability of not making a Type I error on all tests is

None

For a family of C tests, the probability of not making a Type I error for the whole family is

None

For our example, the probability of not making a Type I error on the family

...

  • Loading...
locked icon

Sign in to access this content

Get a 30 day FREE TRIAL

  • Watch videos from a variety of sources bringing classroom topics to life
  • Read modern, diverse business cases
  • Explore hundreds of books and reference titles

Read next

More like this

Sage Recommends

We found other relevant content for you on other Sage platforms.

close

Have you created a personal profile? Login or create a profile so that you can save clips, playlists and searches

close
close
close Signed In to profile You are signed in as:
close
Logged in Institution Institution