Sample Size

The sample size of a survey most typically refers to the number of units that were chosen from which data were gathered. However, sample size can be defined in various ways. There is the designated sample size, which is the number of sample units selected for contact or data collection. There is also the final sample size, which is the number of completed interviews or units for which data are actually collected. The final sample size may be much smaller than the designated sample size if there is considerable nonresponse, [Page 782]ineligibility, or both. Not all the units in the designated sample may need to be processed if productivity in completing interviews is much higher than anticipated to achieve the final sample size. However, this assumes that units have been activated from the designated sample in a random fashion. Survey researchers may also be interested in the sample size for subgroups of the full sample.

In planning to conduct a survey, the survey researcher must decide on the sample design. Sample size is one aspect of the sample design. It is inversely related to the variance, or standard error of survey estimates, and is a determining factor in the cost of the survey. In the simplest situation, the variance is a direct function of the sample size. For example, if a researcher is taking a simple random sample and is interested in estimating a proportion p, then

where n is the sample size. More generally,

where f is the design effect, which reflects the effect of the sample design and weighting on the variance.

In the planning effort for a more complex survey, it is preferable to not focus directly on sample size. It is best to either (a) set a budget and determine the sample size and sample design that minimize the variance for the available budget, or (b) set a desired variance or required reliability, possibly using statistical power analysis, and then determine the sample size and sample design that minimize costs while achieving the desired variance. Sample size does not solely determine either the variance or the cost of a survey and thus is not generally, by itself, a meaningful planning criterion. More useful is effective sample size, which adjusts the sample size for the sample design, weighting, and other aspects of the survey operation.

Even better than fixing the budget and minimizing the variance is to minimize the mean square error (MSE), the researcher can set a desired MSE and minimize the cost to obtain it. MSE is defined as the sum of the variance and the bias squared and thus accounts for more than just the variance. One sample design option may have a larger sample size and a lower variance than a second option but have a larger MSE, and thus it would be a poor choice.

A common misconception is that the needed sample size is a function of the size of the population of interest, or universe. For example, people often think that to achieve a given precision, a much larger sample size is required for a sample of the entire United States than of a large city. This is not generally true. However, if a survey researcher is considering a sample that is a substantial proportion of the population of interest, he or she might apply a finite population adjustment that reduces the variance. Thus, in the case of a sample that is large relative to the population, the needed sample size is reduced. When the researcher is interested in a superpopulation, which is much larger than the actual sampling frame, or is interested in an analytic survey, then the finite population adjustment should not be applied.

...