Natural Experiments

The Scientific Context of Natural Experiments

The value of natural experiments is best appreciated when viewed in the context of other designs. A brief discussion of other designs is therefore illustrative. Observational or naturalistic studies—cross-sectional or longitudinal assessments in which individuals are observed and no experimental influence is brought to bear on them—generally cannot address causal claims. That is because a range of methodological threats, including selection biases and coincidental or spurious associations, undermine causal claims. So, for example, in the developmental and clinical psychology literature, there is considerable interest in understanding the impact of parental mental health—maternal depression is probably the most studied example—on children's physical and mental development. Dozens of studies have addressed this question using a variety of samples and measures. However, almost none of these studies—even large-scale cohort and population studies—are equipped to identify causal mechanisms for several reasons, including (a) genetic transmission is confounded with family processes and other psychosocial risks; (b) maternal depression is virtually always accompanied by other risks that are also reliably linked with children's maladjustment, such as poor parenting and marital conflict; and (c) mate selection for psychiatric disorders means that depressed mothers are more likely to have a partner with a mental illness, which confounds any specific “effect” that investigators might wish to attribute to maternal depression per se. Most of the major risk factors relevant to psychological well-being and public health co-occur; in general terms, risk exposures are not distributed randomly in the population. Indeed, one of the more useful lessons from developmental science has been to demonstrate the ways in which exposures to risk accrue in development.

One response to the problems in selection bias or confounded risk exposure is to address the problem analytically. That is, even if, for example, maternal depression is inherently linked with compromised parenting and family conflict, the “effect” of maternal depression might nevertheless be derived if the confounded variables (compromised parenting and family conflict) are statistically controlled for. There are some problems with that solution, however. If risk processes are confounded in nature, then statistical controlling for one or the other is not a satisfying solution; interpretations of the maternal depression “effect” will be possible but probably not (ecologically) valid. Sampling design strategies to obtain the same kind of leverage, such as sampling families with depressed mothers only if there is an absence of family conflict, will yield an unrepresentative sample of affected families with minimal generalizability. Case-control designs try to gain some leverage over cohort observational studies by tracking a group or groups of individuals, some of whom have a condition(s) of interest. Differences between groups are inferred to be attributable to the condition(s) of interest because the groups were matched on key factors. That is not always possible and the relevant factors to control for are not always known; as a result, between-group and even within-subject variation in these designs is subject to confounders.

...