Subgroup Analysis

Subgroup analysis involves subdividing respondents in a survey into groups on the basis of demographic characteristics (e.g. race, ethnicity, age, education, or gender) or other variables (e.g. party identification, health insurance status, or attitudes toward the death penalty). Analyses of subgroups can be done for a number of reasons. A researcher might analyze differences in variable means or distributions across subgroups to identify disparities or other differences. For example, a researcher studying health care insurance may want to test whether there are differences in the proportion of respondents in different income, education, or race subgroups who are covered by private health care insurance.

Researchers may also want to compare bivariate relationships or multivariate analyses across subgroups to test whether relationships between variables are moderated by subgroup membership. Alternatively, these subgroup comparisons may be conducted to test or establish the generalizability of a relationship or model across subgroups of respondents. For example, a researcher may want to compare the extent to which respondents' pre-election reports of their presidential candidate preferences correspond to their post-election reports of vote choice for respondents who are strong partisans versus those who are weak partisans.

In research involving experimentally manipulated variables, researchers may compare the characteristics of respondents assigned to experimental groups to test whether random assignment has been successful. Researchers also may compare the effect of an experimental manipulation across subgroups to determine if [Page 853]characteristics of respondents moderate the effect of the experimental manipulation or to test or establish the generalizability of the effect across subgroups. For example, survey researchers studying the effects of question order on survey responses could compare the effect of question order for respondents with different levels of education.

For research involving data collected at multiple times (either from the same or different samples of respondents as in longitudinal and panel surveys), subgroup analysis can be used to test whether variable changes over time are the same or different across subgroups. For example, researchers using a panel design to examine changes in math skills in children between the ages of 6 and 10 might compare changes across these ages separately for male and female children.

Subgroup analysis is often used to better understand survey data. However, researchers who intend to use subgroup analysis should keep in mind a number of statistical cautions when using the approach. First, researchers should plan to conduct subgroup analysis in advance (i.e. a priori) rather than deciding to do so after the fact (i.e. post hoc). This helps to address possible concerns with sample size and power. If one or more subgroups are very small, the power to detect effects may be very small, and Type II errors (i.e. concluding there is no difference between groups when there actually is) may be likely. In addition, researchers should be concerned when they are making many subgroup comparisons. Conducting multiple statistical comparisons with the same data increases the chance of Type I error (i.e. concluding there is a difference between groups when the difference is likely due to chance) and researchers conducting subgroup analyses should utilize family-wise error in estimating the significance of their statistical tests to adjust for this possibility.

...