Nonsampling Error

Nonsampling error is a catchall phrase that refers to all types of survey error other than the error associated with sampling. This includes error that comes from problems associated with coverage, measurement, nonresponse, and data processing. Thus, non-sampling error encompasses all forms of bias and variance other than that associated with the imprecision (variance) inherent in any survey sample.

Coverage error refers primarily to the bias and variance that may result when the sampling frame (the list from which the sample is drawn) used to represent the population of interest fails to adequately “cover” the population, and the portion that is missed differs in non-ignorable ways from the portion that is included on the frame. Coverage error also includes bias and variance that can result when a within-unit respondent selection technique is used that does not adequately represent the population at the level of the individual person. Nonresponse error refers to the bias and variance that may result when not all those who are sampled have data gathered from them, and these nonresponders differ in nonignorable ways from responders on variables of interest. Item-level nonresponse error includes the bias and variance that may result from cooperating respondents who do not provide answers (data) to all the variables being measured if the data they would have provided differ in nonignorable ways from the data that the other respondents are providing on those variables. Measurement error refers to bias and variance that may result related to the questionnaire, the behavior of the person who gathers the data, the behavior of respondents, and/or the mode of data collection. Data processing errors refer to the bias and variance that may result from mistakes made while processing data, including the coding and recoding of data, the transformation of data into new variables, the imputation of missing data, the weighting of the data, and the analyses that are performed with data.

Researchers concerned with nonsampling error can take two different strategies to try to deal with it. First, they can implement numerous methodological and other quality control techniques to try to reduce the amount of nonsampling error that results in their studies; this typically adds to the costs of the research study. Second, they can build in methodological studies to try to measure the nature and size of the non-sampling errors that cannot be reduced to negligible levels; this also adds to the project cost but often not as much as the first approach.

Many people appear to think that nonsampling error applies only to research studies that use the survey method of data collection. However, each type of error that makes up nonsampling error has its counterpart in any form of social research, be it qualitative or quantitative, including experiments and quasi-experiments, content analysis, observational research, cognitive interviewing, and focus groups.