Multiplicity Problem

Common Multiple Testing Situations

There are many different settings in which researchers conduct null hypothesis significance testing (NHST). The following are just a few of the more common settings where multiplicity issues arise:

• a.
  conducting pairwise and/or complex contrasts in a linear model with categorical variables;
• b.
  conducting multiple main effect and interaction tests in a factorial analysis of variance (ANOVA) or multiple regression setting;
• c.
  analyzing multiple simple effect, interaction contrast, or simple slope tests when analyzing interactions in linear models;
• d.
  analyzing multiple univariate ANOVAs following a significant multivariate analysis of variance;
• e.
  analyzing multiple correlation coefficients;
• f.
  assessing the significance of multiple factor loadings or factor correlations in factor analysis;
• g.
  analyzing multiple dependent variables separately in linear models;
• h.
  evaluating multiple parameters simultaneously in a structural equation model; and
• i.
  analyzing multiple brain voxels for stimulation in functional MRI research.

Further, as stated previously, most studies involve a mixture of many different types of test statistics.

An important factor in understanding the multiplicity problem is understanding the different ways in which a researcher can “group” their tests of significance. For example, suppose in the study looking at whether student ratings differ across instruction [Page 982]formats that there was also another independent variable, the sex of the instructor. There would now be two “main effect” variables (instruction format and sex of the instructor) and potentially a hypothesis regarding the interaction between instruction format and sex of the instructor. On one hand, the researcher might want to “group” the hypotheses tested under each of the main effect (e.g., pairwise comparisons) and interaction (e.g., simple effect tests) hypotheses into separate “families,” where a family is defined as a group of related hypotheses that are considered simultaneously in the decision process. Therefore, control of the Type I error rate might be imposed separately for each family, or in other words, the Type I error rate for each of the main effect and interaction “families” is maintained at α (the maximum permissible rate of Type I errors for a specific test). On the other hand, the researcher may prefer to treat the entire set of tests for all main effects and interactions as one family, depending on the nature of the analyses and the way in which inferences regarding the results will be made. The central point here is that when researchers conduct multiple tests of significance, they must make important decisions about how their tests are related, and these decisions will directly affect the power and Type I error rates for both the individual tests and for the group of tests conducted in the study.

...