Threats to Validity

Peculiarities of Fisher's Original Approach to Research Design

Two features of Fisher's original context help to explain the initial clarity but long-term difficulties of his approach. These might be called “meta-threats” to validity, because they pertain to the selection of the paradigm of research design validity. First, Fisher developed his paradigm in the context of an agricultural research station, and hence, the things subject to “random assignment” were seeds or soils, not human beings. Second, the station was (U.K.) state run and not for profit. Whereas the former pertains to the concerns originally raised by Donald Campbell and Julian Stanley and in the bulk of this entry, the latter point has received increasing attention—indeed, as part of a general challenge to the idea that statistical significance is a meaningful outcome of hypothesis testing. This latter difficulty is briefly discussed in the next paragraph, before resuming with the threats to research validity specifically related to human subjects.

In the context where Fisher designed experiments on the efficacy of genetic and environmental factors on agricultural output, matters of utility and cost did not figure prominently. A state interested in acquiring a comprehensive understanding of what is likely to make a difference to food production presumed that every hypothesis was equally worthy of study. Thus, “statistical significance” came to be defined as the likelihood of an experimental outcome, given a particular hypothesis, which is tested by seeing whether the outcome would have been the same even if the hypothesis were false (i.e., the null hypothesis). In contrast, someone more explicitly concerned with utility and cost might cast the idea of validity in hypothesis testing as a species of normal rational decision making. In that case, the validity would be defined as follows: Given the available evidence, the cost of accepting a hypothesis vis-à-vis potentially better hypotheses that might require additional testing. Indeed, as Stephen T. Ziliak and Deirdre N. McCloskey observe, this stance was taken by William Sealey Gosset, a student of Karl Pearson who was lab director at Guinness breweries in the early 20th century.

...