Scheffé Test

Background

Like much in statistics, it all started with John Tukey. In a series of oral presentations beginning in about 1950 and culminating in what might be the most cited unpublished manuscript in statistics, Tukey developed the concept of error rates, focusing on the per comparison and familywise rates. Scheffé developed his procedure at the time that Tukey was speaking about his own ideas on multiplicity and, in a footnote to his paper, Scheffé explicitly gave credit to Tukey for the familywise error rate concept that was the basis for his test. The reason for focusing on the familywise error rate as the basis for a test is that it holds the error rate constant at a for the set of all possible contrasts, whether those contrasts had been planned a priori or whether they were reached after examining the data. Scheffé's test was intended specifically as a post hoc test and is most appropriately used that way. It is also a simultaneous inference procedure because intervals or tests are computed simultaneously rather than in a stepwise or layered fashion.

The Test Statistic

Unlike most multiple comparison procedures, Scheffé's test was built on the standard F distribution. Although his derivation of the method is complex, a very clear explanation can be found in the text by Scott Maxwell and Harold Delaney. The basic idea is to find the distribution of the Fmaximum statistic, which is the maximum possible F for any contrast on a set of means. Maxwell and Delaney show that the distribution of Fmaximum is

where k is the number of groups, MSBet and MSerror are the between groups and error mean squares, and N is the total number of observations. The distribution of all possible contrasts (pairwise or complex) will have an Fmaximum distribution under the null hypothesis and can be tested against (k −1)F.05;k-1, N-k· Using that critical value, the familywise error rate will be, at most, α. If the omnibus null hypothesis is rejected, then at least one contrast will equal Fmaximum· Although the Scheffé test is not a protected test like Fisher's least significant difference test, which requires a significant omnibus F before any comparisons are made, there is no point in using Scheffé's test unless the omnibus F is significant. If it is not significant, there can be no significant contrasts. In contrast, just because a significant omnibus F indicates that there will be at least one significant contrast, it does not guarantee that the contrast will be of any interest to researchers.

...