Significance Level

The Probabilities: Alpha and pcalculated or Test Statisticcalculated

Researchers may correctly use statistical significance testing in two general cases: (a) they have a random sample from a population from which an inference is to be drawn, or (b) they believe their sample approximates a random sample. Once this decision is made, the next is to set an alpha level. The setting of the alpha level, a probability ranging between 0 and 1, can be interpreted as the percentage chance of making a sampling error. For instance, an alpha level set at .05 indicates that there is a 5% chance of making an incorrect inference because sampling error creeps into all data without exception, barring collecting data from the entire population. When picking an alpha level, we set a boundary on the probability of making this incorrect inference, called a Type I error. Therefore, alpha is typically set small so that the probability of this error will be low. Thus, this alpha level, also termed pcritical, is selected on the basis of judgment regarding Type I error consequences in any given research situation, guided by personal values regarding these consequences.

After the data are collected and analyzed, part of the output is the second probability, often termed pcalculated. From this point forward, I treat pcalculated and Test Statisticcalculated as being the same because the probabilistic interpretations are similar, and the interpretations are exactly the same. This probability, like the alpha level probability, ranges between 0 and 1 and is calculated based on study parameters. There are two absolutely essential aspects impacting the calculation of the probability pcalculated. First is the assumption that the true population parameters are correctly described in the null hypothesis. This assumption is necessary because the actual population parameters are not known. Therefore, we assume that the null hypothesis is true for all calculations. Second, as the sample size approaches the population size, sampling error decreases, and the statistics calculated from them become more representative. It is intuitive that larger samples are more representative. When we think about this in the opposite direction, small samples are more likely to be comprised of less representative data points, and the statistics calculated for them potentially will be less representative of the population. To illustrate this point, if one were to catch two fish from a pond and both were catfish, one might draw the conclusion that the pond contains only cat-fish. However, with a little more effort and a larger sample, say 100 fish caught, one might learn that the pond ecosystem also consists of bass, perch, gar, and pickerel. So sample size is accounted for in the pcalculated computations. For example, Experiment 1 consists of two groups with 10 members each. The mean for Group 1 is 55 and the mean for Group 2 is 56. The pcalculated will be large and probably exceed the a priori alpha level of p < .05. However, Experiment 2 also consists of two groups (Groups 3 and 4), but this time each group contains 110 members. The mean for Group 3 is exactly the same as Group 1, 55, and the mean for Group 4 is exactly the same as for Group 2, 56 (see Table 1). This time pcalculated is probably small and likely to be less than the a priori alpha level p <.05. Assuming the sample data were randomly selected from the population where the null hypothesis is true, what is the probability of obtaining the sample statistics from the given sample size(s)? The question in both experiments is, “Are the two means statistically significantly different with alpha = .05?” In Experiment 1, the pcalculated = .571, so the means are not statistically different, but in Experiment 2, the pcalculated = .043, so the means are statistically significantly different even though the difference in the [Page 891]means in both experiments is only 1 point (see Table 2). However, the Cohen's d effect size does not change from Experiment 1 to Experiment 2. When pcalculated is less than pcritical or alpha, we use a decision rule that says we will reject the null hypothesis. The decision to reject the null hypothesis is called a statistically significant result. All the decision means is that we believe our sample results are relatively unlikely, given our assumptions, including our assumption that the null hypothesis is exactly true.

...