Post Hoc Test: Scheffe Test

The traditional approach to analysis of variance (ANOVA) relies on the use of an omnibus F statistic to determine whether or not there exists significant difference among the various levels of the variables. The test usually assumes that either one of two conditions (many times both conditions) are true: (1) that there are more than two values for an independent variable (in the case of the one-way ANOVA) or (2) more than one independent variable. The use of the omnibus F statistic intends to protect the system from a Type I error (false positive) that could occur if a scholar simply took the various means for each group and made multiple comparisons using a simple student t statistic.

Each t-test provides a statistic operating independently (assuming a p value that is common, like 5%); the p value (p < .05) becomes not 5% when considered across the entire test of t-tests. What happens is that a set of tests run in a series fails to collectively maintain the original p value (5%). The value of the potential for false positive becomes greater than 5% and that value grows with each successive test. Essentially, the omnibus F statistic intends to provide an overall assessment of the system (often referred to as the family-wise error rate) and the distribution by comparing the relative ratio of between-group variance to within-group variance (the F ratio). The impact of using the F statistic becomes a family-wise error rate that can maintain an overall Type I error rate set at the original value (5%). If the F value is significant, the value indicates at least one significant difference exists among the group means under consideration.

If the F statistic (ratio) is nonsignificant, then one can conclude that no evidence exists for differences between/among the group means. However, if the overall F is significant for an effect, then evidence exists that at least one of the possible comparisons is likely significantly different. In the classic, two-way ANOVA using a 2 × 2 design (two variables, each with two levels, like gender-male/female, level of fear in the message-high/low), significant main effects indicate the means are significantly different. Interpretation of the significant F value when there are only two means is relatively simple, a bit more complex when more than two means/groups are involved.

Suppose in the example of a 2 × 2 design (gender, level of fear) that neither main effect tests is significant; however, a significant interaction exists. The issue is that four means/groups exist in this design (male/high fear, male/low fear, female/high fear, female/low fear). The post hoc test is an examination of which of the four cells are significantly different from other cells. If one thinks of the various comparisons, there exist six possible two-group comparisons with a simple 2 × 2 design. A significant omnibus F statistic indicates at least one those comparisons, possibly all of the comparisons are significantly different. There exist a variety of potential post hoc tests (e.g., Tukey, Duncan, least significant difference); this entry discusses the Scheffe test. Although the tests are highly likely to agree in most circumstances, the tests each highlight or handle different conditions and under some circumstances some tests may be preferred.

...