Clinical Significance

Clinical significance (CS) refers to methodology developed to estimate the meaningfulness of observed [Page 260]change following an intervention (Hansen & Lambert, 1996). It is more stringent than statistical significance, as it requires changes to be both statistically significant and meaningful within a social context.

Traditionally, the outcomes of psychosocial interventions have been evaluated using standard statistical methods, such as t-tests or ANOVAs, to determine whether any observed change from pre- to post-intervention is significant, meaning the change has a magnitude large enough that it is unlikely to have happened by chance. While these tools are useful, they do not address the relevance of any observed change. For instance, the Beck Depression Inventory (BDI) (Beck & Steer, 1993), a common instrument used to measure depressive symptoms, may show a 4-point change from pre- to post-treatment in a sample of patients receiving a treatment for depression. If the sample is large and relatively homogeneous, a 4-point change on the BDI may well be statistically significant. It is unclear, however, what a 4-point difference on the BDI means clinically. Furthermore, a 4-point change on the BDI from a score of 24 to 20 likely has a very different meaning from a 4-point change from a score of 14 to 10. Tests of statistical significance help us understand the probability of a change occurring, but they can't help us understand the practical importance of change.

Clinical significance methodology was developed to address this problem. The foundation was laid by researchers evaluating behavioral modification interventions with children. Standard measurement practices and statistical procedures didn't seem adequate for some of their evaluation needs, so social validation practices were advocated (Kazdin, 1977). Social validation refers to comparing pre- and post-intervention scores against social norms, rather than simply evaluating a change score. For instance, researchers could note the typical level of disruptive behavior of children in a classroom setting, as well as the level of disruptive behavior among children with attention deficit/hyperactivity disorder (ADHD). After a behavioral intervention with children with ADHD, the posttreatment similarity of the ADHD sample to the behavior of a normal classroom could be evaluated, with the idea that a successful intervention will produce behavior that has moved into the “normal” range. CS takes the idea of social validation and formalizes it with clear guidelines that must be met before change can be considered to be “clinically meaningful.”

The process of evaluating the importance of observed change requires normative comparisons, and within clinical significance methodology, this is accomplished by defining independent functional and dysfunctional distributions against which a study sample can be compared (Jacobson & Truax, 1991). After defining appropriate comparison groups, two statistical procedures must be completed. First, a cutoff score is computed, denoting the point at which there is equal probability of a score falling into either the functional or dysfunctional range. A subject beginning treatment in the dysfunctional range of scores must cross this threshold in order to be considered “recovered” after delivery of treatment. Second, a value called the reliable change index (RCI) is computed, based on the variance of the initial, normative groups and the measurement error of the outcome instrument being used. This value is similar to an effect size, corrected for measurement error, and must be surpassed by a participant's change score for that score to be considered reliable. When both of these criteria are met, a participant is considered to have entered the range of functional individuals and have a level of change that is statistically greater than measurement error. This status is considered clinically relevant change and is labeled “recovered” in clinical significance terminology.

...