Critical Difference

Means Model

Multiple comparison tests arise from parametric and nonparametric tests of means, medians, and ranks corresponding to different groups. The parametric case for modeling means can be described

where i = 1 … p (number of treatments) and j = 1 … ni (sample size of the ith treatment). The null hypothesis is that all the means are equal:

and it is tested with an F test. Regardless of the result of this test, a priori tests are always considered. However, post hoc tests are considered only if the F test is significant. Both a priori and post hoc tests compare means and/or linear combinations of means. Comparing means is a natural follow-up to rejecting H0: μ1 = μ2 = … = μp.

Multiple Comparison Tests for Means

Multiple comparison tests serve to uncover which pairs of means or linear contrasts of means are significant. They are often applied to analyze the results of an experiment. When the null hypothesis is that all means are equal, it is natural to compare pairs of means:

The straightforward comparison of the two means can be generalized to a linear contrast:

Linear contrasts describe testing other combinations of means. For example, the researcher might test whether the third treatment mean in an experiment is different from the average of the first and second treatment means,

Table 1 contains equations describing critical differences corresponding to typical multiple comparison tests. In addition to means, there are non-parametric critical differences for medians and for ranks. The number of treatments is denoted by p; the ith treatment group contains ni observations; N = Σ ni denotes the total number of observations in the experiment; α denotes the error rate for each comparison; and k is the number of experimentwise comparisons—where applicable. The experimentwise error rate refers to the overall error rate for the entire experiment. Many of the critical differences in the table can be adjusted to control the experimentwise error rate.

This table is not exhaustive and is intended to be illustrative only. Many of the critical differences have variations.

These same critical differences can be used for constructing confidence intervals. However, caution is warranted as they might not be efficient.

Consequential or Practical Significance

Multiple comparison tests are designed to find statistical significance. Sometimes the researcher's objective is to find consequential significance, which is a special case of statistical significance. Finding statistical significance between two means indicates that they are discernible at a given level of significance, α. Consequential significance indicates that they are discernible and the magnitude of the difference is large enough to generate consequences. The corresponding confusion is pandemic. In order to adjust the critical differences to meet the needs of consequential significance, an extra step is wanted.

...