Sample Size Planning

Sample size planning is the systematic approach to selecting an optimal number of participants to include in a research study so that some specified goal or set of goals can be satisfied. Sample size planning literally addresses the question “What size sample should be used in this study?” but an answer to the question must be based on the particular goal(s) articulated by the researcher. Because of the variety of research questions that can be asked, and the multiple inferences made in many studies, answering this question is not always straightforward. The appropriate sample size depends on the research questions of interest, the statistical model used, the assumptions specified in the sample size planning procedure, and the goal(s) of the study. In fact, for each null hypothesis significance test performed and/or confidence interval constructed, a sample size can be planned so as to satisfy the goals of the researcher (e.g., to reject the null hypothesis and/or obtain a sufficiently narrow confidence interval). Each of the possible sample size planning procedures can suggest a different sample size, and these sample sizes can be very different from one another.

The most common approach when planning an appropriate sample size is the power analytic approach, which has as its goal rejecting a false null hypothesis with some specified probability. Another approach, termed accuracy in parameter estimation (AIPE), has as its goal obtaining a sufficiently narrow (1 −α)% confidence interval for a population parameter of interest, where 1 −α is the desired confidence interval coverage, with α being the Type I error rate. Notice that the two perspectives of sample size planning are fundamentally different in their respective goals, with the former being concerned with rejecting a null hypothesis and the latter with obtaining sufficiently narrow confidence intervals. Perhaps not surprisingly, depending on the specific goals, the implied sample size from the two perspectives can be very different. Although other approaches to sample size planning exist, the power analytic and the AIPE approaches serve as broad categories for conceptualizing the goals of sample size planning.

In null hypothesis significance testing, where an attempt is made to test some null hypothesis, not having an adequate sample size can lead to a failure to reject a false null hypothesis—one that should in fact be rejected. When a false null hypothesis is not rejected, a Type II error is committed. In such situations, the failure to find significance oftentimes renders the study inconclusive because it is still uncertain whether the null hypothesis is true or false. Conversely, if the goal of the research study is to reject some specified null hypothesis, then it is not generally a good use of time and resources to use a larger than necessary sample size. Furthermore, if the goal of the research study is to reject some specified null hypothesis and a power analysis suggests a sample size that would be exceedingly difficult to obtain given available resources, a researcher might decide to conduct a modified version of the originally proposed study [Page 1302]or decide that such a study, at present, would not be a good use of resources because of the low probability of success.

...