Statistical Significance

Probability

Statistical significance testing may be used when working with a random sample from a population, or a sample that is believed to approximate a random, representative sample. Statistical significance testing calls for subjective judgment in establishing a predetermined probability (ranges between 0 and 1.0) of making an inferential error caused by the sampling error. When using statistical significance testing, it requires the use of two forms of probability (P): calculated and critical. Statistical significance is met when P(CALCULATED) is less than P(CRITICAL); when this is the case, the null hypothesis can be “rejected.” Only when the null hypothesis is rejected are the results called “statistically significant.” This simply implies that the sample results are relatively unlikely, given the assumption that the null hypothesis is exactly true.

Misinterpretation of Statistical Significance

When working with statistical significance tests, it is important to be aware of three common misinterpretations. The first misinterpretation is that P(CALCULATED) is an indication of how likely the sample results describe the population. P(CALCULATED) does not inform the replicability of the results. The second misinterpretation is that P(CALCULATED) is the probability that the results were due to chance, and that smaller p values indicate stronger evidence that the null hypothesis is false. This is a misinterpretation because it has been demonstrated that P (CALCULATED) can be smaller than the probability of the null hypothesis being true. The third misinterpretation is that P(CALCULATED) is the probability that the null hypothesis is true. Despite the common misinterpretations, statistical significance testing continues to be widely used even when it does not tell us what we want to know. Scholars suggest that users of statistical significance tests do not fully understand what these tests do, and test users mistakenly believe that statistical significance tests evaluate whether results were due to chance.

...