Meta-Analysis

A single study conducted to examine an issue will not usually establish definitive conclusions. It takes the accumulation of results across studies to begin to establish facts that can be then used to either validate a theory or formulate a new one. For example, many studies have been conducted to examine the relationship between the theoretical concept self-efficacy and academic achievement. That is, does a person's belief in his or her ability to perform academic tasks such as taking tests and completing homework assignments have an impact on academic achievement (e.g., grades, test scores)?

The traditional way to accumulate knowledge across these studies was to conduct a review of the literature. A researcher read all the published studies on the topic and then wrote a narrative describing these studies and an overall set of conclusions. Although these reviews were useful in organizing the studies conducted on a certain topic, the conclusions were based on the subjective impressions of the reviewer. Other concerns were the uneven quality of the studies as well as the difficulty in summarizing some topics in which there were very large numbers of empirical studies, often with what seemed to be conflicting results.

Meta-analysis is an attempt to address the weaknesses of the traditional literature review of empirical studies by using statistical integration and analysis of research findings. Data from studies examining the issue of interest are collected and aggregated, and then statistical tests are conducted to the aggregated or pooled data for the researcher to interpret. Thus, the primary purpose of meta-analysis is two fold, to first summarize the results of empirical research studies and, second, to estimate what the results might have been if all the relevant studies had been conducted without methodological limitations. This second purpose is expected to better reveal the underlying construct-level relationships in which scientists are most interested.

Data are quantified in two important ways in a meta-analysis: (1) The descriptive data from each study are coded; and (2) the results of each study are transformed into an effect size, which is a common metric across studies. This common metric transformation permits the data from different studies to be aggregated and compared, with effect sizes generally weighted to give more emphasis to studies using more participants. Effect sizes are largely sorted into two main types, correlation and standardized mean difference. The product-moment correlation coefficient and its variants provide an overall estimate of the strength of the relationship between two variables. If the focus of the meta-analysis is the effectiveness of some type of treatment or program, then standardized group mean differences are computed to provide an index of effect (e.g., Cohen's d). Research questions for the latter type of explanatory meta-analysis are often in relative terms, such as determining if one type of intervention is better than another type for this particular problem.

In the meta-analysis examining the relationship between self-efficacy and academic achievement, Karen Multon and her colleagues found an overall moderate effect size across all studies, which meant that a significant positive relationship was found between one's beliefs in one's ability to perform academic tasks and one's academic achievement. Additional statistical analysis showed that the variance in reported effect sizes was partially explained by certain study characteristics. For example, there was a stronger overall effect size for low-achieving students than for those students making normative academic progress. Thus, a meta-analysis is conducted not only to compute an overall effect size, but also to examine the relationship between the dependent variable (i.e., effect size) drawn from each study and the independent variables or characteristics of each study (e.g., population attributes, outcome measures, intervention).

...