Behrens-Fisher Test

A common question in statistics involves testing for the equality of two population means, μ1 and μ2, based on independent samples. In many applications, it is reasonable to assume that the population variances, σ21 and σ22, are equal. In this case, the question will usually be addressed by a two-sample t test. The problem of testing for equality of means when the population variances are not assumed to be the same is harder, and is known as the Behrens-Fisher problem.

Suppose we have two samples, x11, x12,…, x1,n and x21, x22,…, x2,n, where the x1i are normally distributed with mean μ1 and variance σ21 and the x2i are normally distributed with mean μ2 and variance σ22, all observations are independent, and it is not assumed that σ21 = σ22. Let X¯i and s2i denote respectively the mean and variance of sample i = 1,2. Now, X¯1 − X¯2 ∼ N(μ1 − μ2, σ21/n1 + σ22/n2) exactly if the original samples are from a normal distribution, and asymptotically if they are not. So, the assumption of normality is not in fact needed.

If we define a pooled variance by

then, with or without the assumption of normality, s2 converges to the same quantity, namely a weighted average of σ21 and σ22,

where w is the limit of the ratio n1/n2 and n1/n2 → w as n1, n2 →∞.

It can be shown that the usual t statistic,

instead of converging to N(0,1) under the null hypothesis of no difference in the population means, converges to a normal distribution with mean 0 and variance (δ+ w)/(δw + 1), where δ is the ratio between σ21 and σ22.

In order to understand the effect of the assumption that δ is not necessarily equal to 1 on inference in this setting, it helps to examine how the expression for the asymptotic variance changes as w and δ vary. It is important to realize that if w = 1, that is, the two sample sizes are equal, either exactly or in the limit, then the asymptotic variance is 1, no matter the value of δ. Thus, with equal sample sizes, inference, at least asymptotically, is not affected by unequal variances. Having nearly equal samples from the two populations thus mitigates the Behrens-Fisher testing problem. Similarly, if the discrepancies in the population variances are not large, such that δ= 1 or nearly so, then we are back in the standard situation, and again asymptotic inference will proceed as before.

The most worrisome situation is when w is small and δ is large. This corresponds to having a much smaller sample from the first population than from the second, when the variance in the first population is much larger than the variance in the second. In this situation, it is necessary to confront the Behrens-Fisher problem directly. A convenient solution, which is only approximate, is to use Welch's t′ statistic, defined as [Page 85]

which acknowledges the fact that the two sample variances cannot be assumed equal. The difference in the procedure derives from the degrees of freedom associated with this test, since it can be shown that,

...