Theory Building

Conceptual Overview

E. Jane Webster and William H. Starbuck investigated the development of theories about organizational behavior over the course of 50 years. They examined the long-term trends in nine relationships that applied psychologists had identified as being very important. The trends showed no progress over time in three relationships, and the explanatory power of five relationships gradually declined toward zero over time. In the most recent data, none of these relationships accounted for more than 5% of the variance in studied phenomena, even though four of the relationships were probably inflated by Hawthorne effects: These four relationships concern the effects of interventions, and because any intervention should yield some effects, the differential impacts of specific interventions would be less than the measures suggest. Only one of the nine relationships suggested significant progress, but almost all of this apparent progress could be attributed to a single early study that had found a weak relationship. Furthermore, what this relationship says might be regarded as trite: Some of the people who say in private that they intend to quit their jobs actually do quit them.

In order for progress to occur, researchers must generate research results that actually contain useful information, and researchers must agree among themselves that they have learned something of lasting value from prior research. Neither of these conditions is being met at present.

Social science researchers have been using statistical significance as an indicator of the importance of their findings, but statistical significance was initially proposed not as a measure of importance but as a measure of the amount of data. Because researchers have been using an inappropriate criterion for success, they have been classifying random noise as statistically significant findings. Many pervasive background correlations affect the population of data that social science researchers gather. These background correlations mean that the expected correlation between two [Page 1544]randomly chosen variables is around 0.09, not zero. Starting with almost any variable, a researcher finds it extremely easy to discover a second variable that correlates at least 0.1 in absolute value with the first variable. In fact, if the researcher were to choose the second variable utterly at random, the odds would be 2 to 1 of coming up with such a variable on the first try, and the odds would be 24 to 1 of discovering such a variable within three tries. In order for a finding to be meaningful, it would have to describe a correlation that is distinctly different from 0.09. However, researchers test their findings against a null hypothesis of zero correlation. Therefore, in typical data, roughly 65% of the correlations are statistically significant at the 5% level. The percentages of statistically significant correlations change considerably with numbers of observations. For studies with more than 180 observations, 72% of the positive correlations and 56% of the negative correlations are statistically significant; whereas for studies with less than 70 observations, 34% of the positive correlations and only 18% of the negative correlations are statistically significant. Thus, positive correlations are noticeably more likely than negative ones to be judged statistically significant.

...