Power

Statistical Significance

The significance level (α) is the probability of rejecting a null hypothesis that is true. The power of the test (1 β) is the probability of correctly rejecting a false null hypothesis. Therefore, as significance levels increase, so does power. One way to increase power is to use a larger significant criterion, which increases the chance of obtaining a statistically significant result (rejecting the null hypothesis). However, doing so increases the risk of obtaining a statistically significant result when the null hypothesis is true (i.e., false positive, or Type I error). A commonly used, albeit arbitrary, significance level is α = .05 which signifies there is a 5% probability that a researcher will incorrectly detect a significant effect when one does not actually exist.

Effect Size

Effect size refers to the magnitude of the effect of interest in the population. Larger effects are easier to detect than small effects. Thus, power to detect a significant effect increases as the magnitude of the effect increases. For example, a researcher may be likely to detect a very large difference between two groups but may have [Page 1066]more difficulty detecting a small difference. In the latter case involving very small effects, the probability of a Type II error, which refers to not finding a significant difference when one actually exists (i.e., a false negative, or β), is high. For instance, if the likelihood of a false negative (β) is .80, and power is equal to 1 β, then power in this case is .20. So, in general, the power to detect small effects is low, although it can be increased by manipulating other parameters, such as sample size.

Sample Size

Sample size refers to the number of observations (n). In the case of human sciences, this often means the number of people involved in the study. In power analysis, the most frequently asked question is how many observations need to be collected in order to achieve sufficient statistical power. When sample size is large, variation within the sample (standard error) becomes smaller and makes standardized effect size larger. Note that there may be times when recommended sample size (resulting from power analysis) will be inadequate (see example below). Although increasing sample size may increase power, there is recognition that too many observations may lead to mistaken detection of trivial effects that are not clinically significant. By contrast, if too few observations are used, a hypothesis test will be weak and less convincing. Accordingly, there may be little chance to detect a meaningful effect even when it exists (Type II error).

...