Content Analysis

Example of Content Analysis in Survey Research

Imagine a questionnaire that asks respondents, What is the biggest problem facing the nation today? Some of the answers that respondents have given to this open-ended question are shown in Figure 1 (along with the spelling and grammar mistakes made by telephone interviewers).

For a survey researcher to be able to analyze the “biggest problem” question, these verbatim answers must be coded into a relatively small and meaningful set of categories. Using the verbatims in Figure 1, a plausible set of categories could be as follows:

• President Bush; His administration and its policies The Republican Congress
[Page 141]
• Honesty in government Immigration; Illegal aliens Moral decline
• Housing
• War in Iraq
• National security; Terrorism
• Misc. Other

Figure 1 Examples of answers given to open-ended question, “What is the biggest problem facing the nation today?”

Coders need to be carefully trained and regularly monitored to apply these categories reliably to each verbatim answer and thereby assign a numerical value to that answer. In this example, a new coded variable would be created that ranges in value from 1 to 9. This variable then could be analyzed via cross-tabulation or other statistical procedures to learn, for example, whether certain demographic characteristics of the respondents (e.g. age, gender, and race) are related to the answers given to the open-ended question.

Content analysis can also be performed by computer software programs. Again, the researchers need to devise a reliable coding scheme in order for the end product to be reliable. For many researchers, the limitations of what software can accomplish are offset by the lower costs of doing the content coding with software compared to the much higher cost of doing it with human coders. However, many content coding solutions will be beyond the capacity of current computer software to apply reliably, and in those instances human coders will need to be utilized.

Analytic Considerations

A general rule of thumb that many survey researchers have found in doing content analyses of open-ended answers is to code as many as three new variables for each open-ended question. For example, if the open-ended question is Q21 in the questionnaire, then the three new variables might be named Q21CAT1, Q21CAT2, and Q21CAT3. This follows from experience that indicates that nearly all respondents will give at least one answer to an open-ended question (since most of these open-ended questions do not ask for only one answer). Many respondents will give two answers, and enough will give three answers to justify coding up to three answers from each respondent. When this approach is used, the researcher also may want to create other new dichotomous (dummy) variables coded “0” or “1” to indicate whether each respondent did or did not mention a certain answer category. Thus, for the earlier example using the “biggest problem” question, new dichotomous variables could be created for each category (BUSH, CONGRESS, HONESTY, IMMIGRATION, etc.). For each of these new variables, the respondent would be assigned the value of “0” if she or he did not mention this category in the open-ended verbatim response and “1” if this category was mentioned.

...