Clinical Significance

Statistical Significance versus Clinical Significance

Statistically significant findings do not always correspond to the client's phenomenological experience or overall evaluation of beneficial impact. First, a research study may have a small effect size yet reveal statistically significant findings due to high power. This typically occurs when a large sample size has been used. Nevertheless, the clinical significance of the research findings may be trivial from the research participants’ perspective. Second, in other situations, a moderate effect size may yield statistically significant results, yet the pragmatic benefit to the client in his or her everyday life is questionable. For example, children diagnosed with attention-deficit/hyperactivity disorder may participate in an intervention study designed to increase concentration and reduce disruptive behavior. The investigators conclude that the active intervention was beneficial on the basis of significant improvement on Connors Ratings Scale scores as well as significantly improved performance on a computerized test measuring sustained attention. The data interpretation is correct from a statistical point of view. However, the majority of parents view the intervention as inconsequential because their children continue to evidence a behavioral problem that disrupts home life. Moreover, most of the treated children see the treatment as “a waste of time” because they are still being teased or ostracized by peers. Despite significant sustained attention performance improvements, classroom teachers also rate the experimental intervention as ineffective because they did not observe meaningful changes in academic performance.

A third scenario is the case of null research findings in which there is also an inconsistency between statistical interpretation and the client or family perspective. For instance, an experimental treatment outcome study is conducted with adult trauma survivors compared with treatment as usual. Overall, the treatment-as-usual group performed superiorly to the new intervention. In fact, on the majority of post-traumatic stress disorder (PTSD) measures, the experimental group evidenced no statistical change from pre- to posttest. Given the additional costs of the experimental intervention, the investigators may decide that it is not worth further investigation. However, qualitative interviews are conducted with the participants. The investigators are surprised to learn that most participants receiving the intervention are highly satisfied although they continue to meet PTSD diagnostic criteria. The interviews demonstrate clinical significance among the participants, who perceive a noticeable reduction in the intensity of daily dissociative symptoms. These participants see the experimental intervention as quite beneficial in terms of facilitating tasks of daily living and improving their quality of life.

...