Pitfalls of Meta-Analyses

    loading  Checking for direct PDF access through Ovid


To the Editor:A meta-analysis is a statistical technique for summarizing and reviewing extant quantitative research. As such, it is ideally suited for obtaining an impression of the efficacy of interventions. Indeed, meta-analyses are among the most highly-cited articles and often provide the basis of treatment recommendations.Unfortunately, meta-analytic procedures are not without limitations. One of the most concerning weaknesses of meta-analyses involving psychotherapy research is related to the quality of the original studies. In fact, a meta-analysis can only be as good as the studies it is based on. A case in point is a recently published meta-analysis by Norton and Price in the present journal (Norton and Price, 2007). The laudable goal of this study was to examine the efficacy of cognitive-behavioral therapy (CBT) across the various anxiety disorders. Their search of the literature yielded 180 trials that met the authors’ inclusion criteria. Because CBT comprises a family of interventions, the authors compared cognitive therapy and exposure therapy alone, in combination, or combined with relaxation training. A number of interesting differences emerged when comparing CBT across the various anxiety disorders diagnoses. For example, CBT appeared to be particularly effective for treating generalized anxiety disorder (GAD).Coincidentally, at around the same time, we (Hofmann and Smits, 2008), independently, performed a meta-analysis with the identical research question in mind: Does the efficacy of CBT vary across the anxiety disorders? In contrast to the meta-analysis by Norton and Price (2007), we decided to include only high-quality, randomized placebo-controlled studies. Of 1165 studies that were initially identified, 27 met our inclusion criteria. Similar to Norton and Price, we found CBT to be a generally effective strategy for treating all anxiety disorders. However, the 2 meta-analyses came to very different answers to the primary study question (i.e., how effective is CBT across the anxiety disorders?). As already mentioned, Norton and Price reported that CBT was particularly effective for generalized anxiety disorder. Interestingly, the effect size for GAD observed in our study was among the weakest relative to the other anxiety disorders. We found the strongest effect sizes for obsessive-compulsive disorder and acute stress disorder. We attribute the difference in the results between the 2 meta-analyses to a number of methodological issues, all of which are of general relevance for meta-analytic procedures and their interpretations. These issues include the following points.(1) The search for studies to be included in a meta-analysis should be comprehensive. Norton and Price limited their search to only peer-reviewed studies written in the English language that appeared in PsycInfo or Medline. Such a search strategy unnecessarily restricts the sample of potentially eligible studies. In fact, of the 27 high-quality studies that were the basis for the Hofmann and Smits (2008) study, only 13 were included in the meta-analysis by Norton and Price.(2) Reasons for exclusion of studies should be theory-based. Consistent with recommendations put forth by the QUORUM group (Moher et al., 1999), we recommend presenting a flow diagram that specifies the reasons for excluding studies.(3) Limiting the inclusion of studies to only high-quality trials may not always be feasible. To allow the reader to make inferences with respect to the outcome of a pooled effect size analyses, it is necessary that the authors report on the quality of the studies that are included in a meta-analysis. We suggest that authors quantify the quality of studies to take differences in quality into consideration when combining their effects. One possibility is to assign each study a quality score based on the Jadad criteria (Jadad et al., 1996).

    loading  Loading Related Articles