Nonsignificance

This entry defines nonsignificance within the context of null hypothesis significance testing (NHST), the dominant scientific statistical method for making inferences about populations based on sample data. Emphasis is placed on the three routes to nonsignificance: a real lack of effect in the population; failure to detect a real effect because of an insufficiently large sample; or failure to detect a real effect because of a methodological flaw. Of greatest importance is the recognition that non-significance is not affirmative evidence of the absence of an effect in the population.

Nonsignificance is the determination in NHST that no statistically significant effect (e.g., correlation, difference between means, and dependence of proportions) can be inferred for a population. NHST typically involves statistical testing (e.g., t test) performed on a sample to infer whether two or more variables are related in a population. Studies often have a high probability of failing to reject a false null hypothesis (i.e., commit a Type II, or false negative, error), thereby returning a nonsignificant result even when an effect is present in the population.

In its most common form, a null hypothesis posits that the means on some measurable variable for two groups are equal to each other. A statistically significant difference would indicate that the probability that a true null hypothesis is erroneously rejected (Type I, or false positive, error) is below some desired threshold (?), which is typically .05. As a result, statistical significance refers to a conclusion that there likely is a difference in the means of the two population groups. In contrast, nonsignificance refers to the finding that the two means do not significantly differ from each other (a failure to reject the null hypothesis). Importantly, nonsignificance does not indicate that the null hypothesis is true, it only indicates that one cannot rule out chance and random variation to explain observed differences. In this sense, NHST is analogous to an American criminal trial, in which there is a presumption of innocence (equality), the burden of proof is on demonstrating guilt (difference), and a failure to convict (reject the null hypothesis) results only in a verdict of “not guilty” (not significant), which does not confer innocence (equality).

Nonsignificant findings might reflect accurately the absence of an effect or might be caused by a research design flaw leading to low statistical power and a Type II error. Statistical power is defined as the probability of detecting an existing effect (rejecting a false null hypothesis) and might [Page 926]be calculated ex ante given the population effect size (or an estimate thereof), the desired significance level (e.g., .05), and the sample size.

Type II errors resulting from insufficient statistical power can result from several factors. First, small samples yield lower power because they are simply less likely than large samples to be representative of the population, and they lead to larger estimates of the standard error. The standard error is estimated as the sample standard deviation divided by the square root of the sample size. Therefore, the smaller the sample, the bigger the estimate of the standard error will be. Because the standard error is the denominator in significance test equations, the bigger it is, the less likely the test statistic will be large enough to reject the null hypothesis. Small samples also contribute to non-significance because sample size (specifically, the degrees of freedom that are derived from it) is an explicit factor in calculations of significance levels (p values). Low power can also result from imprecise measurement, which might result in excessive variance. This too will cause the denominator in the test statistic calculation to be large, thereby underestimating the magnitude of the effect.

...