Data Cleaning

Missing Data

Missing data reduce the sample size available for the analyses. An investigator's research design may require 100 respondents in order to have sufficient power to test the study hypotheses. Substantial effort may be required to recruit and treat 100 respondents. At the end of the study, if there are 10 important variables, with each variable missing only 5% of the time, the investigator may be reduced to 75 respondents with complete data for the analyses. Missing data effectively reduce the power of the study. Missing data can also introduce bias because questions that may be embarrassing or reveal anything illegal may be left blank. For example, if some respondents do not answer items about income, place of birth (for immigrants without documents), or drug use, the remaining cases with complete data are a biased sample that is no longer representative of the population.

Data Errors

Data errors are also costly to the study because lowered reliability attenuates the results. Respondents may make mistakes, and errors can be introduced during data entry. Data errors are more difficult to detect than missing data. Table 1 shows examples of missing data (ethnicity, income), incomplete data (date and place of birth), and erroneous data (sex).

Causes

All measuring instruments are flawed, regardless of whether they are in the physical or social sciences. Even with the best intentions, everyone makes errors. In the social sciences, most measures are self-report. Potentially embarrassing items can result in biased responses. Lack of motivation is also an important source of error. For example, respondents will be highly motivated in high-stakes testing such as the College Board exams but probably do not bring the same keen interest to one's research study.

...