Meta-Analysis: Model Testing

Understanding the Nature of Variability

Normally, in a primary data set (a data set where the sample are a set of participants individually providing information), the typical measures of variability involve standard deviation, range, and variance. The calculation of the measures of dispersion are defined in other encyclopedia entries and involve a determination of how much difference exists between the individual scores and the expected value (mean). The same basic principles apply to the statistics and procedures in a meta-analysis in terms of estimating the basic parameters. What makes the analysis slightly different is the introduction of estimation parameters that must now involve separate data sets.

In a meta-analysis, there are two sets of issues in terms of estimation of sample size: the number of studies or empirical investigations providing estimates used for the calculation of the mean association (k) and the number of units/persons associated with the individual/combined sample size across the set of estimates (N). If all the data were combined into one data set, then the problems of separate samples would become irrelevant. [Page 998]The problem with meta-analysis is that the aggregations or estimations of parameters is not direct, but instead a sense of indirectness becomes an essential part of the process. What happens is that an additional element—the study or data set—becomes a part of the analysis.

With meta-analysis, the set of studies serves as the source of variability. However, the variability or impact of the variability is not identical from study to study. Studies with larger sample sizes are more accurate estimates of the effect than studies with smaller estimates (the same argument holds true if the process weights by contribution to the variance as a form of weighting). The challenge is finding a means to reflect that element of the analysis. This entry does not consider forms of meta-analysis that involve no weighting by either sample size or variance, since much has been written on the implications of not using such procedures.

The implications of whether the average estimate becomes derived using a sample-weighted versus variance-weighted procedure also carries a great set of implications for the analysis. Sample weighting (almost invariably using some random-effects model) creates an average effect and an associated estimate of variance (a number of different procedures for this are possible) whose parameters remain independent. Variance weighting involves the generation of an average whose estimate is no longer independent statistically from an estimation of the variance associated with the average effect. The results of this distinction make the assessment of moderator variables a distinct process.

...