Statistics Commentary Series: Commentary No. 19
1. Which articles to include. As mentioned in the first article, a hallmark of both systematic reviews and MAs is the thoroughness of the search and the inclusion of all articles that meet basic criteria (which may vary from MA to MA). However, the devil is in the details, and in this case, the detail is that small word “all.” Not all studies finally appear in peer-reviewed journals. The results of some research are presented at meetings and never published; others as reports with limited circulation; and still others may have been theses or dissertations. This is referred to as the gray literature (or the “grey” literature in Canada and the U.K.), and the list may also include preprints, technical reports, discussion papers, drafts of articles in the bottom of file drawers, and the like. The question is whether these should be included. Those arguing against their inclusion believe that they have not been published for a very good reason – they are likely methodologically of poorer quality and that if they were sufficiently rigorous, they would have been published; in other words, they remain in the gray literature because they have or would have failed in the peer review process. On the other side of the debate, proponents for their inclusion point to the well-known publication bias mentioned in the previous article that results in mainly articles with positive results finding their way into press. This means that what has been published is a biased sample of all research on a topic, skewing the results in a positive direction. In fact, Kirsch et al. have written a number of papers showing that the effect sizes (ESs) for antidepressants reported in submissions to the Food and Drug Administration but not published were smaller than those in published articles.4,5
Complicating the picture even further is the unfortunate recent growth industry of “predatory journals.” These are journals with very impressive titles (eg, International Journal for Advanced Review and Research in Pharmacy), often published overseas, that promise peer review and free access, but only after authors have paid a hefty fee. In fact, little if no peer review is done. One (of many) tests of the integrity of these journals consisted of a computer-generated paper filled with graphs, footnotes, and utter gibberish, and submitted from a fake institution (the Center for Research in Applied Phrenology; ie, CRAP) – it was accepted.6 Fortunately, a research librarian, Jeffrey Beall, maintains an on-line list of predatory journals7 and publishers.8
So, should articles from the gray literature or predatory journals be included? My personal position is that they should be, but with 2 provisions. First, the quality of the research should be evaluated using 1 of the existing scales, 25 of which were evaluated by Moher et al.9 That allows you to use the score in a meta-regression to determine if the quality of the research affected the ESs.