Methods Box

A methods box is a short news story or sidebar that accompanies poll stories and provides methodological details and clarifications about the survey, including how the respondents were sampled, how the interviews were conducted, the process of weighting the results, and the survey's possible error.

Many newspapers include a boxed feature (sometimes in a smaller type size) alongside a major poll story, and most television networks include the equivalent on their Web sites with details about their survey methodology. For example, it is policy at The New York Times that a methods box accompanies all articles that focus on a New York Times/CBS News Poll or a New York Times Poll.

There are different rules at other papers. The Washington Post and the Los Angeles Times include the basic facts of the survey in the main story with some additional information at the bottom of the accompanying graphic. But an extensive and detailed description of the poll methodology appears on the papers' Web sites.

There are two main criteria in creating a method box. First, it needs to be written in plain English that any layperson can understand. Second, it needs to be statistically correct. It should satisfy the editors that it is literate and the statisticians that it is correct technically. It is not always easy to do both.

Different organizations offer varying levels of detail regarding their methodology, but in general, they adhere to the standards for minimum disclosure that is part of the American Association for Public Opinion Research's code of professional ethics and practices, available on its Web site. The National Council on Public Polls has put together a pamphlet, “Twenty Questions a Journalist Should Ask About Poll Results,” which can be found on its Web site.

Standard elements of methods boxes include interviewing dates, the number and type of respondents, the mode of interviewing, sampling and weighting particulars, and the margin of sampling error. In addition to the method of interviewing, the methodology should also include a list of languages in which data were collected and the name of the company that may have conducted the fieldwork.

Some method boxes describe their margin of sampling error as based on a 95% confidence level. Because “95% confidence level” may not be a concept familiar to the public as a whole, an alternative procedure would be to explain what that actually means. For example, for a survey of approximately 1,000 respondents, The New York Times' methods box reads, “In theory, in 19 cases out of 20, overall results based on such samples will differ by no more than three percentage points in either direction from what would have been obtained by seeking out all American adults.”

If the story involves frequent references to key subgroups, the margin of sampling error for those subgroups should also be included. For example, “For Democrats, the margin of sampling error is plus or minus 4 points and for Republicans it is plus or minus 5 points.” In a pre-election poll, the subsample of “likely voters,” if any, needs to be explained.

...