A randomized controlled study of reviewer bias against an unconventional therapy. Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. Effect of blinding and unmasking on the quality of peer review: a randomized trial. Does exchanging comments of Indian and non-Indian reviewers improve the quality of manuscript reviews? An exploratory study of statistical assessment of papers published in the British Medical Journal. Calling Medical Care reviewers first. A randomized trial. Reviewer bias against the unconventional? A randomized double-blind study of peer review. The effects of blinding on acceptance of research papers by peer review. Effect of written feedback by editors on quality of reviews: two randomized trials. The use of dedicated methodology and statistical reviewers for peer review: a content analysis of comments to authors made by methodology and regular reviewers. Identifying manuscript reviewers: randomized comparison of asking first or just sending. A comparison of authors publishing in two groups of U.S. medical journals. Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. Comparison of review articles published in peer-reviewed and throwaway journals. Readers' evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde. Effects of training on quality of peer review: randomised controlled trial. Evaluating the BMJ guidelines for economic submissions: prospective audit of economic submissions to BMJ and The Lancet. The Medical Journal of Australia Internet peer-review study. Effect of structured workshop training on subsequent performance of journal peer reviewers. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. An intervention to improve the reliability of manuscript reviews for the Journal of the American Academy of Child and Adolescent Psychiatry. The effects of blinding on the quality of peer review. A randomized trial. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. Effect of attendance at a training session on peer reviewer quality and performance. [Effect of statistical review on manuscript quality in Medicina Clínica (Barcelona): a randomized study]. Open peer review: a randomised controlled trial.
A study was designed to test the hypothesis that experts who review papers for publication are prejudiced against an unconventional form of therapy. Two versions were produced (A and B) of a 'short report' that related to treatments of obesity, identical except for the nature of the intervention. Version A related to an orthodox treatment, version B to an unconventional treatment. 398 reviewers were randomized to receive one or the other version for peer review. The primary outcomes were the reviewers' rating of 'importance' on a scale of 1-5 and their verdict regarding rejection or acceptance of the paper. Reviewers were unaware that they were taking part in a study. The overall response rate was 41.7%, and 141 assessment forms were suitable for statistical evaluation. After dichotomization of the rating scale, a significant difference in favour of the orthodox version with an odds ratio of 3.01 (95% confidence interval, 1.03 to 8.25), was found. This observation mirrored that of the visual analogue scale for which the respective medians and interquartile ranges were 67% (51% to 78.5%) for version A and 57% (29.7% to 72.6%) for version B. Reviewers showed a wide range of responses to both versions of the paper, with a significant bias in favour of the orthodox version. Authors of technically good unconventional papers may therefore be at a disadvantage in the peer review process. Yet the effect is probably too small to preclude publication of their work in peer-reviewed orthodox journals. To examine the effect on peer review of asking reviewers to have their identity revealed to the authors of the paper.
                Randomised trial. Consecutive eligible papers were sent to two reviewers who were randomised to have their identity revealed to the authors or to remain anonymous. Editors and authors were blind to the intervention.
                The quality of the reviews was independently rated by two editors and the corresponding author using a validated instrument. Additional outcomes were the time taken to complete the review and the recommendation regarding publication. A questionnaire survey was undertaken of the authors of a cohort of manuscripts submitted for publication to find out their views on open peer review.
                Two editors' assessments were obtained for 113 out of 125 manuscripts, and the corresponding author's assessment was obtained for 105. Reviewers randomised to be asked to be identified were 12% (95% confidence interval 0.2% to 24%) more likely to decline to review than reviewers randomised to remain anonymous (35% v 23%). There was no significant difference in quality (scored on a scale of 1 to 5) between anonymous reviewers (3.06 (SD 0.72)) and identified reviewers (3.09 (0.68)) (P=0.68, 95% confidence interval for difference - 0.19 to 0.12), and no significant difference in the recommendation regarding publication or time taken to review the paper. The editors' quality score for reviews (3.05 (SD 0.70)) was significantly higher than that of authors (2.90 (0.87)) (P<0.005, 95%confidence interval for difference - 0.26 to - 0.03). Most authors were in favour of open peer review.
                Asking reviewers to consent to being identified to the author had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review, but it significantly increased the likelihood of reviewers declining to review. Little research has been conducted into the quality of peer review and, in particular, the effects of blinding peer reviewers to authors' identities or masking peer reviewers' identities.
                To determine whether concealing authors' identities from reviewers (blinding) and/or revealing the reviewer's identity to a coreviewer (unmasking) affects the quality of reviews, the time taken to carry out reviews, and the recommendation regarding publication.
                Randomized trial of 527 consecutive manuscripts submitted to BMJ, which were randomized and each sent to 2 peer reviewers.
                Manuscripts were randomized as to whether the reviewers were unmasked, masked, or uninformed that a study was taking place. Two reviewers for each manuscript were randomized to receive either a blinded or an unblinded version.
                Mean total quality score, time taken to carry out the review, and recommendation regarding publication.
                Of the 527 manuscripts entered into the study, 467 (89%) were successfully randomized and followed up. The mean total quality score was 2.87. There was little or no difference in review quality between the masked and unmasked groups (scores of 2.82 and 2.96, respectively) and between the blinded and unblinded groups (scores of 2.87 and 2.90, respectively). There was no apparent Hawthorne effect. There was also no significant difference between groups in the recommendations regarding publication or time taken to review.
                Blinding and unmasking made no editorially significant difference to review quality, reviewers' recommendations, or time taken to review. Other considerations should guide decisions as to the form of peer review adopted by a journal, and improvements in the quality of peer review should be sought via other means. The quality of peer reviewing in developing countries is thought to be poor. To examine whether this was so, we compared the performance of Indian and non-Indian reviewers who were sent original and review articles submitted to The National Medical Journal of India. We also tested whether informing reviewers that their comments would be exchanged improved the quality of their reviews.
                In a prospective, randomized, blinded study, we sent 100 manuscripts to pairs of peer reviewers (Indian and non-Indian) of which 78 pairs of completed replies were available for analysis. Thirty-eight pairs of reviews were exchanged and 40 were not. The quality of the reviews was assessed by two editors who were unaware of the reviewers' nationality and whether they had been told that their reviews would be exchanged. The quality of the reviews was scored out of 100 (based on a predesigned evaluation proforma). We also measured the time taken to return a manuscript.
                Overall, non-Indian reviewers scored higher than Indians (mean scores non-Indians first, 56.7 v. 48.6, p < 0.001), especially those in the non-exchanged group (58.4 v. 47.3, p < 0.001) but not the exchanged group (54.8 v. 50.0, p < 0.06). Being informed that reviews would be exchanged did not affect the quality of reviews by non-Indians (54.8 exchanged v. 58.4 non-exchanged) or of reviews by Indians (50.0 exchanged v. 47.3 non-exchanged). The editors' assessment of the reviewers matched well (r = 0.59, p < 0.001). Non-Indians took the same amount of time as Indians to return their reviews, although the postage time was at least eight days longer.
                We found that non-Indian peer reviewers were better than Indians and informing them that their views would be exchanged did not seem to affect the quality of their reviews. We suggest that Indian editors should also use non-Indian reviewers and start training programmes to improve the quality of peer reviews in India. Statistical assessment of papers submitted to the British Medical Journal has increased to some 300 papers annually. The assessment produces a recommendation to the editor on each paper from a statistical viewpoint together with a completed checklist that indicates the quality of certain important features. This exploratory study was aimed at monitoring the process. It reports a comparison of checklist answers on 45 papers as originally submitted with those on the papers as subsequently published. Of the 45 papers, only 5 (11%) were considered statistically acceptable at submission, but this increased to 38 (84%) after publication. Revisions had not been made adequately in 4 of the 7 unsatisfactory published papers, and the 3 others were thought to be of dubious validity. A major omission from at least 28 papers was information on sample size calculations. It is concluded that statistical assessment is beneficial but that further efforts by authors and assessors could make it even more effective. nan To test the hypothesis that there is a reviewer bias against publication of a test of an unconventional drug.
                Randomized, controlled, double-blind study of peer review.
                Convenience sample of 291 medical doctors from a wide variety of specialties drawn from a list of conference participants.
                Reviewers were randomly assigned to receive one of two versions of a manuscript. Version M related to an in-vitro experiment on a mainstream drug (Metoprolol). The otherwise identical version V used a highly unconventional drug (beef spleen cell extract) for the same experiment. Reviewers were asked to complete a standardised evaluation sheet including visual analogue scales (VASs) on a set of predefined quality criteria. All participants were debriefed after completion of the study.
                The response rate was 61%. There were no significant differences in VAS ratings between the two versions of the manuscript. Ratings covered the entire range of the VASs.
                In the present setting, there was no evidence for a reviewer-bias against testing an unconventional drug. The low inter-rater reliability, however, suggested inadequate validity of peer review. To study whether reviewers aware of author identity are biased in favor of authors with more previous publications.
                Randomized controlled trial.
                Editorial office of the Journal of Developmental and Behavioral Pediatrics.
                Two "blinded" and two "nonblinded" reviewers assigned to 57 consecutive manuscripts submitted between September 1991 and March 1992.
                Spearman rank correlation coefficients were used to compare the sum of rating scores of 1 to 5 (1, accept; 5, reject) given by the two blinded reviewers, the two nonblinded reviewers, and the editors to the number of articles published previously by the first and senior authors (as determined from requested curricula vitae). Blinded reviewers were sent a questionnaire asking whether they could determine the identity of the authors, how they knew, and whether they thought binding changed the quality or difficulty of their review.
                The Wilcoxon Sign Rank Test disclosed no differences between blinded and nonblinded scores. The number of previous articles by the senior author was significantly correlated (P < .01) with blinded scores (r = -.45) and editors' decisions (r = -.45), but not with nonblinded scores; the number of articles by the first author was correlated (P < .05) with editors' decisions (r = -.35) but not with blinded or nonblinded scores. Fifty (46%) of 108 blinded reviewers correctly guessed the identity of the authors, mostly from self-references and knowledge of the work; 86% believed blinding did not change the quality of their review, and 73% believed it did not change the difficulty of performing a review.
                Blinded reviewers and editors in this study, but not nonblinded reviewers, gave better scores to authors with more previous articles. These results suggest that blinded reviewers may provide more unbiased reviews and that nonblinded reviewers may be affected by various types of bias. Better peer review is needed, but proven methods to improve quality are unknown. Our objective was to determine whether written feedback to reviewers improves subsequent reviews.
                Eligible reviewers were randomized to intervention or control (receiving other reviewers' unscored reviews and the editor's decision letter). Study 1 (September 1998-September 2000) included reviewers with a median quality score of 3 or lower; study 2 (April 2000-January 2002), reviewers with median score of 4 or lower. Study 1 was designed with a power of 0.80 to detect a difference in score of 1; study 2, with a power of 0.80 to detect a difference of 0.5. All reviewers were at a peer-reviewed journal (Annals of Emergency Medicine). The main outcome measure was the editor's routine quality rating (1-5) of all reviews (blinded to study enrollment).
                For study 1, 51 reviewers were eligible and randomized and 35 had sufficient data (182 reviews) for analysis. The mean individual reviewer rating change was 0.16 (95% confidence interval [CI], -0.26 to 0.58) for control and -0.13 (-0.49 to 0.23) for intervention. For study 2, 127 reviewers were eligible and randomized, and 95 had sufficient data (324 reviews). Controls had a mean individual rating change of 0.12 (95% CI, -0.20 to 0.26) and intervention reviewers, 0.06 (-0.19 to 0.31).
                In study 1, minimal feedback from editors on review quality had no effect on subsequent performance of poor-quality reviewers, and the trend was toward a negative effect. In study 2, feedback to average reviewers was more extensive and supportive but produced no improvement in reviewer performance. Simple written feedback to reviewers seems to be an ineffective educational tool. In 1997, Annals of Emergency Medicine initiated a protocol by which every original research article, in addition to each regular review, was concurrently evaluated by 1 of 2 methodology and statistical reviewers. We characterized and contrasted comments made by the methodology and regular peer reviewers.
                After pilot testing, interrater reliability assessment, and revision, we finalized a 99-item taxonomy of reviewer comments organized in 8 categories. Two authors, uninvolved in the writing of reviews, classified each comment from a random sample of methodology reviews from 1999. For 30 of these reviews (15 for each methodology reviewer), the 2 authors also scored all (range 2 to 5) regular reviews.
                Sixty-five reviews by methodologist A, 60 by methodologist B, and 68 by regular reviewers were analyzed. Comments by methodologist A most frequently concerned the presentation of results (33% of all comments) and methods (17%). Methodologist B commented most frequently on presentation of results (28%) and statistical methods (16%). Regular reviewers most frequently made non-methodology/statistical comments (45%) and comments on presentation of results (18%). Of note, comments made by methodology and regular reviewers about methods issues were often contradictory.
                The distributions of comments made by the 2 methodology and statistical reviewers were similar, although reviewer A emphasized presentation and reviewer B stressed statistical issues. The regular reviewers (most of whom were unaware that a dedicated methodology and statistical reviewer would be reviewing the article) paid much less attention to methodology issues. The 2 dedicated methodology and statistical reviewers created reviews that were similarly focused and emphasized methodology issues that were distinct from the issues raised by regular reviewers. Some journals routinely query potential referees before sending manuscripts ("askfirst"), whereas others just send manuscripts and allow referees to opt out ("justsend"). It is not known which protocol results in more completed reviews or shorter review time.
                Trial to assess proportion of referee turndowns and length of review process, conducted at editorial office of Obstetrics & Gynecology and involving 283 consecutive qualifying manuscripts. For each, a referee was randomly assigned to askfirst (manuscript sent only after affirmative response within 3 days) and another to justsend (manuscript sent with request to review; could opt out).
                Only 64% of askfirst referees assented initially (15% declined [vs 8% for justsend, P =.008] and 21% failed to respond within 3 working days, necessitating a replacement). But once manuscript was mailed, mean time to file a review was significantly shorter for askfirst (21.0 vs 25.0 days, P<.001); thus, overall time to receipt of review did not differ significantly (24.7 vs 25.9 days, P =.19), nor did review quality (P =.39).
                Askfirst led to a higher rate of referee turndown than did justsend, but assenting askfirst referees completed reviews faster. The overall time for the review process did not differ between the 2 protocols. This study compared the editorial peer review experiences of authors who published in two groups of indexed U.S. medical journals. The study tested the hypothesis that after one journal rejects a manuscript an author selects a less well-known journal for submission. Group One journals were defined as those indexed in 1992 MEDLINE that satisfied several additional qualitative measures; Group Two journals were indexed in the 1992 MEDLINE only. Surveys were sent to the first authors of 616 randomly selected articles, and 479 surveys were returned, for a response rate of 78.1%. A total of 20.8% of Group One and 15.7% of Group Two articles previously had been rejected. Group One authors were more likely to select a journal for its prestige and article quality, while Group Two authors were more likely to have been invited to submit the manuscript. More than 60% of both groups felt the peer review had offered constructive suggestions, but that it had changed article conclusions less than 3% of the time. Both groups thought the review process only marginally improved content, organization, or statistical analysis, or clarified conclusions. Between 3% and 15% of all authors received considerable conflicting advice from different reviewers. Authors from both groups differed as to their reasons for journal selection, their connections with the publishing journal, and patterns of resubmission after rejection. All authors may not be equal in the eyes of reviewers. Specifically, well-known authors may receive less objective (poorer quality) reviews. One study at a single journal found a small improvement in review quality when reviewers were masked to author identity.
                To determine whether masking reviewers to author identity is generally associated with higher quality of review at biomedical journals, and to determine the success of routine masking techniques.
                A randomized controlled trial performed on external reviews of manuscripts submitted to Annals of Emergency Medicine, Annals of Internal Medicine, JAMA, Obstetrics & Gynecology, and Ophthalmology.
                Two peers reviewed each manuscript. In one study arm, both peer reviewers received the manuscript according to usual masking practice. In the other arm, one reviewer was randomized to receive a manuscript with author identity masked, and the other reviewer received an unmasked manuscript.
                Review quality on a 5-point Likert scale as judged by manuscript author and editor. A difference of 0.5 or greater was considered important.
                A total of 118 manuscripts were randomized, 26 to usual practice and 92 to intervention. In the intervention arm, editor quality assessment was complete for 77 (84%) of 92 manuscripts. Author quality assessment was complete on 40 (54%) of 74 manuscripts. Authors and editors perceived no significant difference in quality between masked (mean difference, 0.1; 95% confidence interval [CI], -0.2 to 0.4) and unmasked (mean difference, -0.1; 95% CI, -0.5 to 0.4) reviews. We also found no difference in the degree to which the review influenced the editorial decision (mean difference, -0.1; 95% CI,-0.3 to 0.3). Masking was often unsuccessful (overall, 68% successfully masked; 95% CI, 58%-77%), although 1 journal had significantly better masking success than others (90% successfully masked; 95% CI, 73%-98%). Manuscripts by generally known authors were less likely to be successfully masked (odds ratio, 0.3; 95% CI, 0.1-0.8). When analysis was restricted to manuscripts that were successfully masked, review quality as assessed by editors and authors still did not differ.
                Masking reviewers to author identity as commonly practiced does not improve quality of reviews. Since manuscripts of well-known authors are more difficult to mask, and those manuscripts may be more likely to benefit from masking, the inability to mask reviewers to the identity of well-known authors may have contributed to the lack of effect. To compare the quality, presentation, readability, and clinical relevance of review articles published in peer-reviewed and "throwaway" journals.
                We reviewed articles that focused on the diagnosis or treatment of a medical condition published between January 1 and December 31, 1998, in the 5 leading peer-reviewed general medical journals and high-circulation throwaway journals. Reviewers independently assessed the methodologic and reporting quality, and evaluated each article's presentation and readability. Clinical relevance was evaluated independently by 6 physicians.
                Of the 394 articles in our sample, 16 (4.1%) were peer-reviewed systematic reviews, 135 (34.3%) were peer-reviewed nonsystematic reviews, and 243 (61.7%) were nonsystematic reviews published in throwaway journals. The mean (SD) quality scores were highest for peer-reviewed articles (0.94 [0.09] for systematic reviews and 0.30 [0.19] for nonsystematic reviews) compared with throwaway journal articles (0.23 [0.03], F(2,391) = 280.8, P<.001). Throwaway journal articles used more tables (P =.02), figures (P =.01), photographs (P<.001), color (P<.001), and larger font sizes (P<.001) compared with peer-reviewed articles. Readability scores were more often in the college or higher range for peer-reviewed journals compared with the throwaway journal articles (104 [77.0%] vs 156 [64.2%]; P =.01). Peer-reviewed article titles were judged less relevant to clinical practice than throwaway journal article titles (P<.001).
                Although lower in methodologic and reporting quality, review articles published in throwaway journals have characteristics that appeal to physician readers. Academic biomedical journals use peer review and editing to help to select and improve the quality of articles. We have investigated whether articles accepted by the Nederlands Tijdschrift voor Geneeskunde, the Dutch Journal of Medicine, were improved after peer review and editing (post-acceptance scientific and copy editing).
                400 readers of the journal (100 each of medical students, recent medical graduates, general practitioners, and specialists) were invited to participate in a questionnaire survey. The first 25 from each group who agreed to participate were included. We posted a pack containing a set of identically appearing typescripts (ie, blinding) of the submitted, accepted, and published versions of 50 articles that had been published in Ned Tijdschr Geneeskd. Each evaluator received two of the sets of versions, and each set was evaluated by one person from each group. The package also included two questionnaires: the first was used to compare the submitted with the accepted version (25 questions), the second compared the accepted with the published version (17 questions). The questions were answered on five-point scales, and were about the quality of the articles or were general/overall scores. We analysed the data as scores of 3-5 (ie, improvement) versus 1-2.
                After peer review, the quality in 14 of 23 questions (61%) was significantly improved (p = 0.03 or smaller). In particular, the overall score and general medical value were significantly improved (p = 0.00001 for each). Editing led to significant improvement in 11 of 16 questions (69%, p = 0.017 or smaller), and especially in style and readability (p = 0.001 and p = 0.004). Generally, we found no differences between the scores of the four categories of evaluators. 72% of the evaluators correctly identified which version was which.
                Evaluations by readers of the Ned Tijdschr Geneeskd indicated significant improvement of published articles after both peer review and editing. We think that peer review and editing are worthwhile tasks. We also think that possible biases would have had a negligible effect on our results (including the fact that we selected the first 25 evaluators who responded, that some evaluators may have read the published version, and that one questionnaire may have looked more scientific than the other, more editorial one). To determine the effects of training on the quality of peer review.
                Single blind randomised controlled trial with two intervention groups receiving different types of training plus a control group.
                Reviewers at a general medical journal. Interventions Attendance at a training workshop or reception of a self taught training package focusing on what editors want from reviewers and how to critically appraise randomised controlled trials.
                Quality of reviews of three manuscripts sent to reviewers at four to six monthly intervals, evaluated using the validated review quality instrument; number of deliberate major errors identified; time taken to review the manuscripts; proportion recommending rejection of the manuscripts.
                Reviewers in the self taught group scored higher in review quality after training than did the control group (score 2.85 v 2.56; difference 0.29, 95% confidence interval 0.14 to 0.44; P = 0.001), but the difference was not of editorial significance and was not maintained in the long term. Both intervention groups identified significantly more major errors after training than did the control group (3.14 and 2.96 v 2.13; P < 0.001), and this remained significant after the reviewers' performance at baseline assessment was taken into account. The evidence for benefit of training was no longer apparent on further testing six months after the interventions. Training had no impact on the time taken to review the papers but was associated with an increased likelihood of recommending rejection (92% and 84% v 76%; P = 0.002).
                Short training packages have only a slight impact on the quality of peer review. The value of longer interventions needs to be assessed. Editorial management of articles on health economics may benefit from guidelines for peer review and revision.
                To assess whether publication (in August 1996) of the BMJ guidelines on peer review of economics submissions made any difference to editorial and peer review processes, quality of submitted manuscripts, and quality of published manuscripts.
                Before-after study conducted in the editorial offices of BMJ and The Lancet of the effect of the BMJ guidelines on review and revision of economics submissions, defined as those making explicit comments about resource allocation and/or costs of interventions.
                Editorial fate and changes in the quality of submissions.
                A total of 2982 manuscripts were submitted to the 2 journals during the before periods, 105 (3.5%) of which were economics submissions. Of these, 27 (24.3%) were full economics evaluations, and 78 (75.7%) were other economics submissions. Overall acceptance rate was 11.6% (12/105). During the after period 2077 manuscripts were submitted to the 2 journals, 87 (4.2%) of which were economics submissions. Eighteen (20.7%) were full economics evaluations, and 69 (79.3%) were other economics submissions. Overall acceptance rate was 6.9% (6/87). Although a number of manuscripts could not be traced to determine whether they were economics submissions, there appeared to be little difference between the 2 journals in numbers or editorial fate of the manuscripts. There was no change in the quality of submitted manuscripts, but BMJ editors found the guidelines and checklists useful and sent fewer economics submissions for external peer review in the after phase.
                Publication of the guidelines helped the BMJ editors improve the efficiency of the editorial process but had no impact on the quality of economics evaluations submitted or published. Peer review of medical papers is a confidential consultancy between the reviewer and the journal editor, and has been criticised for its potential bias and inadequacy. We explored the potential of the internet for open peer review to see whether this approach improved the quality and outcome of peer review.
                Research and review articles that had been accepted for publication in The Medical Journal of Australia (MJA) were published together with the reviewers' reports on the worldwide web, with the consent of authors and referees. Selected readers' e-mailed comments were electronically published as additional commentary; authors could reply or revise their paper in response to readers' comments. Articles were edited and published in print after this open review.
                60 (81%) of 74 authors agreed to take part in the study, together with 150 (92%) of 162 reviewers. There was no significant difference in the performance of commissioned reviewers before and during the study. Four articles were not included because of insufficient time before print publication. Of the remaining 56 papers, 28 received 52 comments from 42 readers (2% of readers submitted comments). Most readers' comments were short and specific, and seven articles were changed by the authors in response.
                Open peer review is acceptable to most authors and reviewers. Postpublication review by readers on the internet is no substitute for commissioned prepublication review, but can provide editors with valuable input from individuals who would not otherwise be consulted. Readers also gain insight into the processes of peer review and publication. Study objective: We sought to determine whether peer reviewers who attend a formal interactive training session produce better reviews.
                Peer reviewers were invited to attend a formal, 4-hour, highly interactive workshop on peer review. Attendees received a sample manuscript to read and review in writing in advance. The workshop included presentations on analyzing a study and the journal's expectations for a quality review, discussion of the sample manuscript's flaws and how to address them in a review, discussion of the reviews written by the attendees, and discussion of real reviews of other manuscripts illustrating key points. The performance of attendees on the basis of standard editor quality ratings (1 to 5) was assessed for the 2 years after workshop attendance. Control reviewers matched for previous review quality and volume were selected from nonattendees of the workshop. In study 1, all average reviewers received a standard written invitation. In study 2, 75 randomly selected average reviewers were personally and actively recruited with intensive follow-up by means of e-mail and telephone calls in an effort to reduce self-selection bias.
                In study 1, 25 reviewers volunteered for the course, were eligible for study, attended, and were compared with 25 matched control reviewers. Of attendees filling out evaluations, 19% thought it somewhat and 81% thought it very helpful. All thought it would improve their subsequent reviews, and 85% thought it would improve their review ratings. The mean change in rating after the workshop was 0.11 (95% confidence interval [CI] -0.25 to 0.48) for control reviewers and 0.10 (95% CI -0.20 to 0.39) for attendees. In study 2, of 75 reviewers intensively recruited, only 12 (41%) of those who said they would attend did. All of the participants thought the workshop would improve their performance and ratings. Test scores at the end of the workshop improved in 73% of participants compared with scores on pretests. The control reviewers' average rating changed by -0.10 (95% CI -0.49 to 0.29) versus 0.06 (95% CI -0.34 to 0.23) for attendees.
                Among invited peer reviewers, voluntary attendance at a highly structured and interactive workshop was low and did not improve the quality of subsequent reviews, contrary to the predictions of attendees. Efforts to aggressively recruit average reviewers to a second workshop were time consuming, had low success rates, and showed a similar lack of effect on ratings, despite improvement in scores on a test instrument. Workshop teaching formats, although traditional, are of unproven efficacy. To evaluate the effects of peer review and editing on manuscript quality.
                Editorial offices of Annals of Internal Medicine.
                Masked before-after study. MANUSCRIPTS: 111 consecutive original research manuscripts accepted for publication at Annals between March 1992 and March 1993.
                We used a manuscript quality assessment tool of 34 items to evaluate the quality of the research report, not the quality of the research itself. Each item was scored on a 1 to 5 scale. Forty-four expert assessors unaware of the design or aims of the study evaluated the manuscripts, with different persons evaluating the two versions of each manuscript (before and after the editorial process).
                33 of the 34 items changed in the direction of improvement, with the largest improvements seen in the discussion of study limitations, generalizations, use of confidence intervals, and the tone of conclusions. Overall, the percentage of items scored three or more increased by an absolute 7.3% (95% CI, 3.3% to 11.3%) from a baseline of 75%. The average item score improved by 0.23 points (CI, 0.07 to 0.39) from a baseline mean of 3.5. Manuscripts rated in the bottom 50% showed two- to threefold larger improvements than those in the top 50%, after correction for regression to the mean.
                Peer review and editing improve the quality of medical research reporting, particularly in those areas that readers rely on most heavily to decide on the importance and generalizability of the findings. The effects of methods used to improve the interrater reliability of reviewers' ratings of manuscripts submitted to the Journal of the American Academy of Child and Adolescent Psychiatry were studied.
                Reviewers' ratings of consecutive manuscripts submitted over approximately 1 year were first analyzed; 296 pairs of ratings were studied. Intraclass correlations and confidence intervals for the correlations were computed for the two main ratings by which reviewers quantified the quality of the article: a 1-10 overall quality rating and a recommendation for acceptance or rejection with four possibilities along that continuum. Modifications were then introduced, including a multi-item rating scale and two training manuals to accompany it. Over the next year, 272 more articles were rated, and reliabilities were computed for the new scale and for the scales previously used.
                The intraclass correlation of the most reliable rating before the intervention was 0.27; the reliability of the new rating procedure was 0.43. The difference between these two was significant. The reliability for the new rating scale was in the fair to good range, and it became even better when the ratings of the two reviewers were averaged and the reliability stepped up by the Spearman-Brown formula. The new rating scale had excellent internal consistency and correlated highly with other quality ratings.
                The data confirm that the reliability of ratings of scientific articles may be improved by increasing the number of rating scale points, eliciting ratings of separate, concrete items rather than a global judgment, using training manuals, and averaging the scores of multiple reviewers. Peer reviewers are blinded sometimes to authors' and institutions' names, but the effects of blinding on review quality are not known. We, therefore, conducted a randomized trial of blinded peer review. Each of 127 consecutive manuscripts of original research that were submitted to the Journal of General Internal Medicine were sent to two external reviewers, one of whom was randomly selected to receive a manuscript with the authors' and institutions' names removed. Reviewers were asked, but not required, to sign their reviews. Blinding was successful for 73% of reviewers. Quality of reviews was higher for the blinded manuscripts (3.5 vs 3.1 on a 5-point scale). Forty-three percent of reviewers signed their reviews, and blinding did not affect the proportion who signed. There was no association between signing and quality. Our study shows that, in our setting, blinding improves the quality of reviews and that research on the effects of peer review is possible. Anxiety about bias, lack of accountability, and poor quality of peer review has led to questions about the imbalance in anonymity between reviewers and authors.
                To evaluate the effect on the quality of peer review of blinding reviewers to the authors' identities and requiring reviewers to sign their reports.
                Randomized controlled trial.
                A general medical journal.
                A total of 420 reviewers from the journal's database.
                We modified a paper accepted for publication introducing 8 areas of weakness. Reviewers were randomly allocated to 5 groups. Groups 1 and 2 received manuscripts from which the authors' names and affiliations had been removed, while groups 3 and 4 were aware of the authors' identities. Groups 1 and 3 were asked to sign their reports, while groups 2 and 4 were asked to return their reports unsigned. The fifth group was sent the paper in the usual manner of the journal, with authors' identities revealed and a request to comment anonymously. Group 5 differed from group 4 only in that its members were unaware that they were taking part in a study.
                The number of weaknesses in the paper that were commented on by the reviewers.
                Reports were received from 221 reviewers (53%). The mean number of weaknesses commented on was 2 (1.7, 2.1, 1.8, and 1.9 for groups 1, 2, 3, and 4 and 5 combined, respectively). There were no statistically significant differences between groups in their performance. Reviewers who were blinded to authors' dentities were less likely to recommend rejection than those who were aware of the authors' identities (odds ratio, 0.5; 95% confidence interval, 0.3-1.0).
                Neither blinding reviewers to the authors and origin of the paper nor requiring them to sign their reports had any effect on rate of detection of errors. Such measures are unlikely to improve the quality of peer review reports. To determine whether attendance at a voluntary training workshop improves quality ratings of medical journal peer reviewers.
                Peer reviewers for Annals of Emergency Medicine who completed two or more reviews during the 20 months before or the 20 months after October 1995 were eligible. Reviews were routinely rated by editors on a subjective 5-point quality scale. Comparisons were made between reviewers who chose to attend a 4-hour workshop on peer review sponsored by the journal in 1995 (attendees) and 2 groups of reviewers who did not attend: controls matched for review quality and number of reviews completed before the workshop, and unmatched controls. Guest reviewers were excluded.
                A total of 298 reviewers completed 1906 reviews before the workshop and 2,194 after the workshop; 2,117 of these reviews were rated by editors. Forty-five attendees participated in the workshop, 39 of whom had sufficient ratings for analysis. Matched controls were almost identical in performance to attendees, but unmatched controls had performed fewer reviews and had lower average ratings before the workshop. There was no significant change in any performance measurement after the workshop, including average quality rating, percent change in quality rating, odds ratio for recommending acceptance, and odds ratio for congruence with editor's decision.
                In a self-selected group of experienced reviewers who attended a 4-hour workshop on peer review, no effect could be identified in subsequent performance as measured by editors' quality ratings or reviewer performance statistics. The statistical review of biomedical articles should result in an improved quality. The objective of this study was to compare the effects of clinical review and joint clinical and statistical review on manuscript quality, in articles submitted to Medicina Clínica (Barcelona), a Spanish weekly journal of internal medicine.
                Original papers arriving between May 2000 and February 2001 were randomized either to a clinical review group or a clinical and statistical review group. Two evaluators, blinded to the paper's group, assessed the quality improvement in both groups, from submission to publication using a modified version ot the Goodman et al. scale. The protocol required that final versions arrived before the end of May 2001.
                Final sample size was 43 manuscripts, evaluated before and after peer review. On the intention to treat analysis, the estimated effect of statistical review was 1.35 (95% CI: -0.45 to 3.16) positive, but not statistically significant. The analysis of the reviewers' comments revealed some protocol deviations. Taking into account the spontaneous inclusion of statistical experts in the clinical group, the estimated effect was statistically significant, with a confidence interval of 0.3 to 3.7.
                The inclusion of a statistical expert in the peer review process improves manuscript quality, although in the intention to treat analysis the improvement was not statistically significant. Most scientific journals practise anonymous peer review. There is no evidence, however, that this is any better than an open system.
                To evaluate the feasibility of an open peer review system.
                Reviewers for the British Journal of Psychiatry were asked whether they would agree to have their name revealed to the authors whose papers they review; 408 manuscripts assigned to reviewers who agreed were randomised to signed or unsigned groups. We measured review quality, tone, recommendation for publication and time taken to complete each review.
                A total of 245 reviewers (76%) agreed to sign. Signed reviews were of higher quality, were more courteous and took longer to complete than unsigned reviews. Reviewers who signed were more likely to recommend publication.
                This study supports the feasibility of an open peer review system and identifies such a system's potential drawbacks.
At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research. However, the methodological problems in studying peer review are many and complex. At present, the absence of evidence on efficacy and effectiveness cannot be interpreted as evidence of their absence. A large, well-funded programme of research on the effects of editorial peer review should be urgently launched.