Funnel Plot

Publication Bias

Studies across many academic disciplines have shown that research that yields nonsignificant results or negative results is less likely to make it through the peer-review process and be published. Thus, meta-analyses that rely on published articles are likely to yield biased results, overestimating the true effect size and underestimating the variability of effect sizes. Because it is more difficult to find results of unpublished studies, this may be true even if conference papers and technical reports are included in the sample of studies used to calculate the weighted mean effect size.

Statistical Tests for Asymmetry in Funnel Plots

Interpretation based on visual representation is subjective and concerns have been raised whether researchers can readily identify significant asymmetry in funnel plots. Based on these concerns, quantitative approaches to identifying unexpected findings have been developed. One such approach is Colin Begg and Madhuchhanda Mazumdar’s (1994) rank correlation nonparametric test between intervention effect estimate and its variance. Others are Matthias Egger and colleagues’ (1997) regression method, also known as a formal test for asymmetry in funnel graph for continuously measured intervention effect estimate, and Jaime Peters, Roger Harbord, or Gerta Rücker’s tests for log odds ratio effect sizes. First, Begg and Mazumdar’s rank correlation approach is used to examine the relationship between standardized effect size and the variances (or standard errors). Begg and Mazumdar’s test is similar to Maurice Kendall’s rank correlation test (Kendall’s tau), which is based on comparing the ranks of standardized effect size and variance. The ranks are the ordered values by standardized effect sizes; for example, the largest value of standardized effect size is ranked as 1, the next largest standardized effect size is ranked as 2, and so on. Tau is interpreted like a correlation that zero shows no relationship between effect size and precision, and correlations different from zero show a relationship. Second, Egger’s linear regression method tests when the standard normal deviate is regressed against its precision, which is defined as the inverse of the standard error. This method is different from Begg and Mazumdar’s test. Egger uses the actual values of the effect sizes and their precision, but Begg and Mazumdar’s method uses the rank rather than actual values. Also, the power of Egger’s method is usually higher than power for the rank correlation test. For the third approach, Peter, Harbord, or Rücker’s tests are used to avoid the relationship between the log odds ratio and its standard error when there are fairly large intervention effects. Begg and Mazumdar’s test and Egger’s test are used for continuous outcomes when intervention effects are measured as mean differences. However, Peter, Harbord, or Rücker’s tests are used for dichotomous outcomes when intervention [Page 601]effects are measured as odds ratio. Recommendations are given for the tests for funnel plot asymmetry. The tests are generally used when there are more than 10 studies in the meta-analysis because test power is usually too low to differentiate chance from real asymmetry when there are less than 10 studies. Tests for funnel plot asymmetry should not be used when the study sizes are similar. Suggested statistical tests are widely used to determine objectivity of appraisal while controlling for Type I error, that is, to find publication bias when there is no bias. The other possibility for the evaluation of funnel plot asymmetry error is Type II error, which is failure to identify publication bias.

...