Post Hoc Analysis

Post hoc analysis applies to tests of differences among sample statistics when the specific hypothesis to be tested has been suggested by the values of the statistics themselves. Perhaps the most common statistic to be tested is the sample mean in experiments involving three or more means. In such cases, it is most common to begin with the testing of an analysis of variance (ANOVA) F test. A nonsignificant F test implies that the full null hypothesis of the equality of all population means is plausible. Consequently, no additional testing would be considered.

Suppose a significant ANOVA F test is found in an experiment involving k = 4 means. If the last [Page 1057]two means are greater than the first two means, a researcher might want to know if that difference would also be found in comparing the corresponding population means. That researcher would be interested in testing the hypothesis

Equivalent hypotheses would be

or

where ψrepresents the ordered coefficients (1, 1, −1, −1) applied to the corresponding four population means. In general, any contrast among four means could be expressed by the four ordered coefficients (c1, c2,c3, c4).

To test the hypothesis using the four sample means, ψis estimated by applying the four coefficients to the corresponding sample means to calculate Y as

For equal sample sizes with common value, N,a corresponding sum of squares is obtained by

For unequal sample sizes, N1,N2, …, Nkthe fomula is

The contrast has df = 1, so dividing by l (i.e., keeping the same value) changes the SSyto a mean square, MSy. If the contrast had been chosen without first examining the sample means (especially if it was chosen before the experiment was performed), then it would be an a priori contrast rather than a post hoc contrast. In that case, an Ftest for the contrast, Fy,is calculated by

where MSerroris the denominator of the AN OVA Ftest. The critical value would be obtained from an Fdistribution with 1 and dferrordegrees of freedom.

For post hoc analysis, Henry Scheffe provided a similar method for testing the contrast hypothesis. For a contrast among kmeans, the MSyis

The critical value is obtained from an Fdistribution with k 1and dferrordegrees of freedom. If the critical value is determined with a significance level, athe probability of one or more Type I errors will be limited to ano matter how many contrasts are evaluated as post hoc contrasts by the Scheffe procedure. In fact, Scheffe even proved that if the ANOVA Ftest is significant, then at least one post hoc contrast must be significant by the Scheffé procedure.

Scheffé's procedure can be used for testing hypotheses about any number of means in a contrast. However, pairwise testing of only two means at a time can be done with more powerful procedures. John Tukey proposed a single, critical difference, CD, for all pairs of means in a group of kmeans each with sample size N. That value is

where q1-α{k,dferror)is the 100(1 α)percentage point of the Studentized range distribution with parameters kand dferror. Tukey's procedure limits the probability of one or more Type I errors to αeven when testing all k(k −1)/2 pairs of kmeans and even if the ANOVA Ftest is notsignificant. However, it is almost a universal practice to apply Tukey's procedure only after a significant ANOVA Ftest.

...