Critical Difference

Means Model

Multiple comparison tests arise from parametric and nonparametric tests of means, medians, and ranks corresponding to different groups. The parametric case for modeling means can be described as $y_{ij}={\mu }_i+{\epsilon }_{ij},$ which often assumes ${\epsilon }_{ij}\stackrel{\text{IID}}{~}N\left(o,{\sigma }^2\right),$

where i = 1 . . . p (number of treatments) and j = 1 . . . ni (sample size of the ith treatment). The null hypothesis is that all of the means are equal:

$$H_0={\mu }_1={\mu }_2=\cdots ={\mu }_p,$$

$$H_A:{\mu }_i\ne {\mu }_j;i\ne j,$$

and the appropriate test is an F test. Regardless of the result of this test, a priori tests are always considered. However, post hoc tests are considered only if the F test is significant. That is, there are at least two means that are significantly different. Both a priori and post hoc tests compare means and/or linear combinations of means. Comparing means is a natural follow-up after rejecting $H_0:{\mu }_1={\mu }_2=\dots ={\mu }_p$, to infer which pairs of means are different.

Multiple Comparison Tests for Means

Multiple comparison tests serve to uncover which pairs of means or linear contrasts of means are significantly different. They are often applied to analyze the results of an experiment. When the null hypothesis is that all means are equal, it is natural to compare pairs of means:

$$H_0:{\mu }_i={\mu }_j,$$

$$H_A:{\mu }_i\ne {\mu }_j;i\ne j.$$

The straightforward comparison of the two means can be generalized to a linear contrast:

$$H_0:\sum c_i{\mu }_i=0,$$

$$H_A:\sum c_i{\mu }_i\ne 0.$$

Linear contrasts describe testing other combinations of means. For example, the researcher might test whether the third treatment mean in an experiment is different from the average of the first and second treatment means, ${\mu }_3\ne \left({\mu }_1+{\mu }_2\right)/2$. This can be expressed as

$$H_0:{\mu }_3-\frac{1}{2}{\mu }_1-\frac{1}{2}{\mu }_2=0,$$

$$H_A:{\mu }_3-\frac{1}{2}{\mu }_1-\frac{1}{2}{\mu }_2\ne 0.$$

Table 1 contains equations describing critical differences corresponding to typical multiple comparison tests. The first term (often called a critical value) is a distance in terms of standard errors for the comparison, and the second term is an estimate of that standard error. The number of treatments is denoted by p; k is the number of comparisons; the ith treatment group contains ni observations; $N={\displaystyle \sum n_i}$ denotes the total number of observations in the experiment; α denotes the experimentwise error rate (this is sometimes an upper bound and cannot always be calculated exactly); and ${\alpha }_k=1-{(1-\alpha )}^{k-1}$. The comparisonwise error rate refers to the error rate for all of the comparisons, and the experimentwise error rate refers to the error rate for the entire experiment. The degrees of freedom used in the table, N − p, are for a one-way treatment structure. This list is not exhaustive and is intended to be illustrative only. Many of the critical differences have variations. These same critical differences can be used for constructing confidence intervals. However, caution is warranted as the intervals might not be efficient. In addition to means, there are nonparametric critical differences for medians and for ranks.

...