How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. What should be included in meta-analyses? An exploration of methodological issues using the ISPOT meta-analyses. Publication bias and meta-analyses: a practical example. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses?
nan To explore the impact of methodologic issues on the results of meta-analyses. The following issues were examined: the type of literature search strategy used; inclusion or exclusion of non-peer-reviewed studies; the inclusion or exclusion of non-English language publications; the effect of trial quality; and the inclusion or exclusion of non-placebo-controlled studies.
                The International Study of Perioperative Transfusion (ISPOT) meta-analyses were used to evaluate each of the methodologic issues. The 10 meta-analyses consisted of technologies to reduce the need for perioperative red blood cell transfusion. The number of trials for each of the meta-analyses varied from 2 to 45. Both EMBASE and MEDLINE searches were conducted, including the use of systematic search strategies.
                MEDLINE identified the vast majority of trials. Alone, MEDLINE would have missed 8 studies compared to 10 for EMBASE. Use of the systematic search strategies greatly reduced the number of articles to be reviewed compared to open searches. Type of publication, country of study origin, inclusion of non-English publications, and trial quality had very little impact on the estimates of effect. The use of placebo versus open-label control affected the magnitude of the odds ratio for two of the meta-analyses. The results of the two meta-analyses were not statistically significant if only placebo-controlled trials were included.
                While methodologic issues had very little impact on the ISPOT meta-analyses, further studies are needed in a variety of other clinical settings. Because MEDLINE, coupled with a review of the references in the identified trials, identified the vast majority of trials, one needs to consider the costs and benefits of searching EMBASE and the pursuance of unpublished and unindexed trials. Publication bias is widely appreciated, but considerable time and effort are needed to locate and obtain data from unpublished randomized controlled trials (RCTs), those published in non-English language journals or those reported in the gray literature; for this publication, we will call this collection of trials the "gray+literature." However, excluding such trials from systematic reviews could introduce bias and give rise to misleading conclusions.
                We aimed to explore and quantify the impact of inclusion of gray+ literature on the results of all completed individual patient data (IPD) reviews coordinated by our group (13 meta-analyses). For each IPD review, results were calculated for RCTs fully published in English language journals and RCTs fully published in English language journals and the gray+literature.
                The IPD meta-analyses based only on RCTs that were fully published in English language journals tended to give more favorable results than those that included RCTs from the gray+literature. Although in most cases the addition of gray+data gave less encouraging results, moving the estimated treatment effect toward a null result, the direction of effect was not always predictable.
                We recommend that all systematic reviews should at least attempt to identify trials reported in the gray+literature and, where possible, obtain data from them. The inclusion of only a subset of all available evidence in a meta-analysis may introduce biases and threaten its validity; this is particularly likely if the subset of included studies differ from those not included, which may be the case for published and grey literature (unpublished studies, with limited distribution). We set out to examine whether exclusion of grey literature, compared with its inclusion in meta-analysis, provides different estimates of the effectiveness of interventions assessed in randomised trials.
                From a random sample of 135 meta-analyses, we identified and retrieved 33 publications that included both grey and published primary studies. The 33 publications contributed 41 separate meta-analyses from several disease areas. General characteristics of the meta-analyses and associated studies and outcome data at the trial level were collected. We explored the effects of the inclusion of grey literature on the quantitative results using logistic-regression analyses.
                33% of the meta-analyses were found to include some form of grey literature. The grey literature, when included, accounts for between 4.5% and 75% of the studies in a meta-analysis. On average, published work, compared with grey literature, yielded significantly larger estimates of the intervention effect by 15% (ratio of odds ratios=1.15 [95% CI 1.04-1.28]). Excluding abstracts from the analysis further compounded the exaggeration (1.33 [1.10-1.60]).
                The exclusion of grey literature from meta-analyses can lead to exaggerated estimates of intervention effectiveness. In general, meta-analysts should attempt to identify, retrieve, and include all reports, grey and published, that meet predefined inclusion criteria.
This review shows that published trials tend to be larger and show an overall greater treatment effect than grey trials. This has important implications for reviewers who need to ensure they identify grey trials, in order to minimise the risk of introducing bias into their review.