Propensity-Weighted Web Survey

History

Propensity scoring has traditionally been applied in biostatistics to estimate causal effects. Harris Interactive, a New York-based Web survey business, first applied propensity scoring to correct for selection bias in Web surveys. Harris Interactive uses special “Webographic” questions as propensity variables. Webographic questions—also called “lifestyle,” “atti-tudinal,” or “psychographic” questions—are meant to capture the difference between the online and the offline population.

The Practice of Propensity Scoring

In practice, the Web sample and the reference sample are combined to form a single data set. An indicator variable, indicating whether a respondent is from the Web sample, is regressed on the propensity questions, usually via logistic regression, representing the probability that respondents within certain characteristics are from the Web sample. The propensity scores are computed as the predicted values from this logistic regression. The propensity scores can be used in a variety of ways, including as weights for stratification and matching techniques (research). Using the inverse propensity scores as adjustment weights is appealing because the concept of reweighting is familiar and because standard survey software can be used to conduct statistical tests. Another popular method is to stratify the propensity scores into quintiles and to use the resulting five-level categorical variable in a post-stratification step.

Limitations

First, it is not possible to adjust for unbalanced, unobserved variables that correlate with outcomes unless the unobserved variable strongly correlates with observed variables. Second, to calibrate the Web survey, the propensity scoring approach requires a reference sample. For example, Harris Interactive conducts regular RDD phone surveys for that purpose. This requirement currently limits the appeal of this method and makes it most useful for panel surveys. Propensity scoring attempts to achieve balance. That is, after the propensity weighting adjustment, the distribution of the propensity variables should be the same for both the Web sample and the reference sample. The traditional goal to find a logistic regression model that fits the data well is therefore not necessarily useful. Instead, the researcher should verify whether balance was achieved. Preliminary research seems to indicate that the propensity adjustment reduces the bias considerably but does not remove it altogether for all outcome variables. One direction for future research is to find out which set of propensity variables works best for which set of outcomes. Of additional note, propensity scoring has also been applied to adjust for nonresponse bias when data are available for both nonrespondents and respondents of a survey.

...