Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Randomization is important in studies with pain outcomes: systematic review of transcutaneous electrical nerve stimulation in acute postoperative pain. Meta-analysis of home-care effects on mortality and nursing-home placement. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Epistemology and experimentation: an examination of quality factors in research design. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. How study design affects outcomes in comparisons of therapy. II: Surgical. How study design affects outcomes in comparisons of therapy. I: Medical. Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy. Funding source, trial outcome and reporting quality: are they related? Results of a pilot study. Impact of study quality on outcome in placebo-controlled trials of homeopathy. Random versus nonrandom assignment in controlled experiments: do you get the same answer? Impact of random assignment on study outcome: an empirical examination.
To explore whether reported methodologic quality affects estimated intervention effects in randomized trials and contributes to discrepancies between the results of large randomized trials and small randomized trials in meta-analyses.
                Meta-analyses of randomized trials that included at least one large trial (>/=1000 participants) were included, regardless of the therapeutic area. Eligible meta-analyses were identified through electronic searches and bibliographies of relevant articles.
                Full-length randomized trials.
                Methodologic quality was assessed according to reported randomization, double blinding, and follow-up as separate components and by using the Jadad composite scale.
                Fourteen meta-analyses involving 190 randomized trials from eight therapeutic areas were included. Compared with large trials, intervention effects were exaggerated in small trials with inadequate allocation sequence generation (ratio of odds ratios, 0.46 [95% CI, 0.25 to 0.83]; P = 0.011), inadequate allocation concealment (ratio of odds ratios, 0.49 [CI, 0.27 to 0.86]; P = 0.014), and no double blinding (ratio of odds ratios, 0.52 [CI, 0.28 to 0.96]; P = 0.01). Large trials did not differ significantly from small trials with adequate generation of the allocation sequence, adequate allocation concealment, or adequate double blinding. No association was seen between reported follow-up and intervention effects. The Jadad scale provided no additional information because the scale and the quality components overlapped substantially.
                Inadequate generation of the allocation sequence, allocation concealment, and double blinding lead to exaggerated estimates of intervention benefit and may contribute to discrepancies between the results of large randomized trials and small randomized trials in meta-analyses. We set out to examine the evidence for the importance of randomization of transcutaneous electrical nerve stimulation (TENS) in acute postoperative pain. Controlled studies were sought; randomization and analgesic and adverse effect outcomes were summarized. Forty-six reports were identified by searching strategies. Seventeen reports with 786 patients could be regarded unequivocally as randomized controlled trials (RCT) in acute postoperative pain. No meta-analysis was possible. In 15 of 17 RCT, we judged there to be no benefit of TENS compared with placebo. Of the 29 excluded trials, 19 had pain outcomes but were not RCT; in 17 of these 19 TENS studies, the authors concluded that TENS had a positive analgesic effect. No adverse effects were reported. Non-randomized studies overestimated treatment effects. This meta-analysis was designed to illustrate how the relatively new discipline of meta-analysis could be employed in health-services research. In pursuit of this goal, the results of 13 studies on the effect of home care on mortality and nursing-home placements were examined. The analysis demonstrated a small, beneficial effect of home care on mortality, which fell short of statistical significance. Study-to-study variation in the odds ratio was found that was not strongly associated with differences in the study samples, designs, or interventions as categorized here. The analysis produced stronger evidence of a reduction in nursing-home placements. In this case, differences in study design explained much of the heterogeneity in results, with the randomized controlled trials showing considerably weaker effects. The analysis also provides insight into the benefits and limitations of alternative meta-analytic methods. The "odd man out" method recently described by Walker et al. detected statistically significant heterogeneity of effects across studies for nursing-home placement, but not for mortality. Loglinear modeling made it easier to detect and explore study-to-study variation in home-care effects. Few meta-analyses of randomised trials assess the quality of the studies included. Yet there is increasing evidence that trial quality can affect estimates of intervention efficacy. We investigated whether different methods of quality assessment provide different estimates of intervention efficacy evaluated in randomised controlled trials (RCTs).
                We randomly selected 11 meta-analyses that involved 127 RCTs on the efficacy of interventions used for circulatory and digestive diseases, mental health, and pregnancy and childbirth. We replicated all the meta-analyses using published data from the primary studies. The quality of reporting of all 127 clinical trials was assessed by means of component and scale approaches. To explore the effects of quality on the quantitative results, we examined the effects of different methods of incorporating quality scores (sensitivity analysis and quality weights) on the results of the meta-analyses.
                The quality of trials was low. Masked assessments provided significantly higher scores than unmasked assessments (mean 2.74 [SD 1.10] vs 2.55 [1.20]). Low-quality trials (score < or = 2), compared with high-quality trials (score > 2), were associated with an increased estimate of benefit of 34% (ratio of odds ratios [ROR] 0.66 [95% CI 0.52-0.83]). Trials that used inadequate allocation concealment, compared with those that used adequate methods, were also associated with an increased estimate of benefit (37%; ROR=0.63 [0.45-0.88]). The average treatment benefit was 39% (odds ratio [OR] 0.61 [0.57-0.65]) for all trials, 52% (OR 0.48 [0.43-0.54]) for low-quality trials, and 29% (OR 0.71 [0.65-0.77]) for high-quality trials. Use of all the trial scores as quality weights reduced the effects to 35% (OR 0.65 [0.59-0.71]) and resulted in the least statistical heterogeneity.
                Studies of low methodological quality in which the estimate of quality is incorporated into the meta-analyses can alter the interpretation of the benefit of intervention, whether a scale or component approach is used in the assessment of trial quality. Forty-four research articles published in the occupational therapy literature were examined to determine the effect of design quality on study outcome. All 44 of the studies examined involved a parallel-groups comparison design. Twenty-two of the studies included random assignment of subjects to various groups, and the remaining 22 investigations used some nonrandom method to determine subject allocation. A standardized metric (i.e., effect size) was used to determine the effect of the independent variable in the 44 studies. The data analysis revealed that effect-size values were not significantly different between studies that involved random assignment and those not involving random assignment. The argument is made that such design characteristics as random assignment should be examined as moderator variables in any attempt to synthesize findings from multiple studies. Such an approach would treat the design used in a study as one of many possible variables that could influence the outcome. This approach would modify the a priori assumption that one research design is inherently superior to another regardless of the research question or context. nan Conventional reviews of research on the efficacy of psychological, educational, and behavioral treatments often find considerable variation in outcome among studies and, as a consequence, fail to reach firm conclusions about the overall effectiveness of the interventions in question. In contrast meta-analytic reviews show a strong, dramatic pattern of positive overall effects that cannot readily be explained as artifacts of meta-analytic technique or generalized placebo effects. Moreover, the effects are not so small that they can be dismissed as lacking practical or clinical significance. Although meta-analysis has limitations, there are good reasons to believe that its results are more credible than those of conventional reviews and to conclude that well-developed psychological, educational, and behavioral treatment is generally efficacious. Specific features of trial quality may be associated with exaggeration or shrinking of the observed treatment effect in randomized studies. Therefore, assessment of trial quality is often used in meta-analysis. However, the degree to which specific quality measures are associated with treatment effects has not been well established across a broad range of clinical areas.
                To determine if quality measures are associated with treatment effect size in randomized controlled trials (RCTs).
                Quality measures from published quality assessment scales were evaluated in RCTs included in meta-analyses from 4 medical areas (cardiovascular disease, infectious disease, pediatrics, and surgery). Included meta-analyses incorporated at least 6 RCTs, examined dichotomous outcomes, and demonstrated significant between-study heterogeneity in the odds ratio (OR) scale.
                Relative ORs comparing overall treatment effect (summary OR) of high vs low-quality studies, as determined by each quality measure, with relative ORs less than 1 indicating larger treatment effect in low-quality studies.
                Twenty-four quality measures were analyzed for 276 RCTs from 26 meta-analyses. Relative ORs of high vs low-quality studies for these quality measures ranged from 0.83 to 1.26; none was statistically significantly associated with treatment effect. The proportion of studies fulfilling specific quality measures varied widely in the 4 medical areas. In analyses limited to specific medical areas, placebo control, multicenter studies, study country, caregiver blinding, and statistical methods were significantly associated with treatment effect on 7 occasions. These relative ORs ranged from 0.40 to 1.74. However, the directions of these associations were not consistent.
                Individual quality measures are not reliably associated with the strength of treatment effect across studies and medical areas. Although use of specific quality measures may be appropriate in specific well-defined areas in which there is pertinent evidence, findings of associations with treatment effect cannot be generalized to all clinical areas or meta-analyses. To determine if inadequate approaches to randomized controlled trial design and execution are associated with evidence of bias in estimating treatment effects.
                An observational study in which we assessed the methodological quality of 250 controlled trials from 33 meta-analyses and then analyzed, using multiple logistic regression models, the associations between those assessments and estimated treatment effects.
                Meta-analyses from the Cochrane Pregnancy and Childbirth Database.
                The associations between estimates of treatment effects and inadequate allocation concealment, exclusions after randomization, and lack of double-blinding.
                Compared with trials in which authors reported adequately concealed treatment allocation, trials in which concealment was either inadequate or unclear (did not report or incompletely reported a concealment approach) yielded larger estimates of treatment effects (P < .001). Odds ratios were exaggerated by 41% for inadequately concealed trials and by 30% for unclearly concealed trials (adjusted for other aspects of quality). Trials in which participants had been excluded after randomization did not yield larger estimates of effects, but that lack of association may be due to incomplete reporting. Trials that were not double-blind also yielded larger estimates of effects (P = .01), with odds ratios being exaggerated by 17%.
                This study provides empirical evidence that inadequate methodological approaches in controlled trials, particularly those representing poor allocation concealment, are associated with bias. Readers of trial reports should be wary of these pitfalls, and investigators must improve their design, execution, and reporting of trials. We analysed the results of 221 comparisons of an innovation with a standard treatment in surgery published in six leading surgery journals in 1983 to relate features of study design to the magnitude of gain. For each comparison we measured the gain attributed to the innovation over the standard therapy by the Mann-Whitney statistic and the difference in proportion of treatment successes. For primary treatments (aimed at curing or ameliorating a patient's principal disease), an average gain of 0.56 was produced by 20 randomized controlled trials. This was less than the 0.62 average for four non-randomized controlled trials, 0.63 for 19 externally controlled trials, and 0.57 for 73 record reviews (0.50 represents a toss-up between innovation and standard). For secondary therapies (used to prevent or treat complications of therapy), the average gain was 0.53 for 61 randomized controlled trials, 0.58 for eleven non-randomized controlled trials, 0.54 for eight externally controlled trials, and 0.55 for 18 record reviews. Readers of studies evaluating new treatments, particularly for primary treatments, may consider adjustment of the gain according to the study type. We analysed 113 reports published in 1980 in a sample of medical journals to relate features of study design to the magnitude of gains attributed to new therapies over old. Overall we rated 87 per cent of new therapies as improvements over standard therapies. The mean gain (measured by the Mann-Whitney statistic) was relatively constant across study designs, except for non-randomized controlled trials with sequential assignment to therapy, which showed a significantly higher likelihood that a patient would do better on the innovation than on standard therapy (p = 0.004). Randomized controlled trials that did not use a double-blind design had a higher likelihood of showing a gain for the innovation than did double-blind trials (p = 0.02). Any evaluation of an innovation may include both bias and the true efficacy of the new therapy, therefore we may consider making adjustments for the average bias associated with a study design. When interpreting an evaluation of a new therapy, readers should consider the impact of the following average adjustments to the Mann-Whitney statistic: for trials with non-random sequential assignment a decrease of 0.15, for non-double-blind randomized controlled trials a decrease of 0.11. Homoeopathy is widely used, but specific effects of homoeopathic remedies seem implausible. Bias in the conduct and reporting of trials is a possible explanation for positive findings of trials of both homoeopathy and conventional medicine. We analysed trials of homoeopathy and conventional medicine and estimated treatment effects in trials least likely to be affected by bias.
                Placebo-controlled trials of homoeopathy were identified by a comprehensive literature search, which covered 19 electronic databases, reference lists of relevant papers, and contacts with experts. Trials in conventional medicine matched to homoeopathy trials for disorder and type of outcome were randomly selected from the Cochrane Controlled Trials Register (issue 1, 2003). Data were extracted in duplicate and outcomes coded so that odds ratios below 1 indicated benefit. Trials described as double-blind, with adequate randomisation, were assumed to be of higher methodological quality. Bias effects were examined in funnel plots and meta-regression models.
                110 homoeopathy trials and 110 matched conventional-medicine trials were analysed. The median study size was 65 participants (range ten to 1573). 21 homoeopathy trials (19%) and nine (8%) conventional-medicine trials were of higher quality. In both groups, smaller trials and those of lower quality showed more beneficial treatment effects than larger and higher-quality trials. When the analysis was restricted to large trials of higher quality, the odds ratio was 0.88 (95% CI 0.65-1.19) for homoeopathy (eight trials) and 0.58 (0.39-0.85) for conventional medicine (six trials).
                Biases are present in placebo-controlled trials of both homoeopathy and conventional medicine. When account was taken for these biases in the analysis, there was weak evidence for a specific effect of homoeopathic remedies, but strong evidence for specific effects of conventional interventions. This finding is compatible with the notion that the clinical effects of homoeopathy are placebo effects. There has been increasing concern regarding the potential effects of the commercialization of research.
                In order to examine the relationships between funding source, trial outcome and reporting quality, recent issues of five peer-reviewed, high impact factor, general medical journals were hand-searched to identify a sample of 100 randomized controlled trials (20 trials/journal). Relevant data, including funding source (industry/not-for-profit/mixed/not reported) and statistical significance of primary outcome (favouring new treatment/favouring conventional treatment/neutral/unclear), were abstracted. Quality scores were assigned using the Jadad scale and the adequacy of allocation concealment.
                Sixty-six percent of trials received some industry funding. Trial outcome was not associated with funding source (p=.461). There was a preponderance of favourable statistical conclusions among published trials with 67% reporting results that favored a new treatment whereas 6% favoured the conventional treatment. Quality scores were not associated with funding source or trial outcome.
                It is not known whether the absence of significant associations between funding source, trial outcome and reporting quality reflects a true absence of an association or is an artefact of inadequate statistical power, reliance on voluntary disclosure of funding information, a focus on trials recently published in the top medical journals, or some combination thereof. Continued and expanded monitoring of potential conflicts is recommended, particularly in light of new guidelines for disclosure that have been endorsed by the ICMJE. We investigated the influence of indicators of methodological quality on study outcome in a set of 89 placebo-controlled clinical trials of homoeopathy in three different ways: (1) The results of studies meeting single criteria (explicit statement of random allocation, allocation concealment, double-blinding, completeness of follow-up) of methodological quality were compared with those of studies not meeting the criteria in univariate and multivariate analyses; (2) The results of studies scoring above and below predefined scores in two quality assessment scales were compared; (3) Primary studies were consecutively entered into a cumulative meta-analysis according to the summary scores derived from the quality assessment scales. All analyses were performed using meta-regression methods. Studies that were explicitly randomized and were double-blind as well as studies scoring above the cut-points yielded significantly less positive results than studies not meeting the criteria. In the cumulative meta-analyses, there was a trend for increasing effect sizes when more studies with lower-quality scores were added. However, there was no linear relationship between quality scores and study outcome. We conclude that in the study set investigated, there was clear evidence that studies with better methodological quality tended to yield less positive results. Because summarizing disparate study features into a single score is problematic, meta-regression methods simultaneously investigating the influence of single study features seem the best method for investigating the impact of study quality on outcome. Psychotherapy meta-analyses commonly combine results from controlled experiments that use random and nonrandom assignment without examining whether the 2 methods give the same answer. Results from this article call this practice into question. With the use of outcome studies of marital and family therapy, 64 experiments using random assignment yielded consistently higher mean post-test effects and less variable posttest effects than 36 studies using nonrandom assignment. This difference was reduced by about half by taking into account various covariates, especially pretest effect size levels and various characteristics of control groups. The importance of this finding depends on (a) whether one is discussing meta-analysis or primary experiments, (b) how precise an answer is desired, and (c) whether some adjustment to the data from studies using nonrandom assignment is possible. It is concluded that studies using nonrandom assignment may produce acceptable approximations to results from randomized experiments under some circumstances but that reliance on results from randomized experiments as the gold standard is still well founded. Sixty research investigations published in the biomedical literature were analyzed to examine the effect of design attributes on outcome. All 60 studies included a controlled trial involving a pretest, a therapeutic intervention, and a posttest across at least two groups. Thirty of the trials used random assignment of participants to treatment or control conditions and 30 trials employed some nonrandom method of subject assignment. Trial results were aggregated and evaluated by comparing effect sizes for the primary statistical test of the hypothesis. Data analysis revealed that the trial results, as measured by effect size, did not vary across therapeutic trials using random assignment versus those using nonrandom allocation of subjects. The impact of design attributes in the interpretation of multiple clinical trials addressing a similar research question is examined. The argument is made that various design attributes frequently associated with methodological quality should be considered as important moderator variables and their influence on trial outcome should not be assumed a priori but rather examined empirically.
The results of randomised and non-randomised studies sometimes differed. In some instances non-randomised studies yielded larger estimates of effect and in other instances randomised trials yielded larger estimates of effect. The results of controlled trials with adequate and inadequate/unclear concealment of allocation sometimes differed. When differences occurred, most often trials with inadequate or unclear allocation concealment yielded larger estimates of effects relative to controlled trials with adequate allocation concealment. However, it is not generally possible to predict the magnitude, or even the direction, of possible selection biases and consequent distortions of treatment effects from studies with non-random allocation or controlled trials with inadequate or unclear allocation concealment.