Evaluation of cardiovascular grant-in-aid applications by peer review: influence of internal and external reviewers and committees. Peer review: a simplified approach. Quantitative evaluation of peer review of program project and center applications in allergy and immunology. Rating intervals: an experiment in peer review. Chance and consensus in peer review. Peer review: inter-reviewer agreement during evaluation of research grant applications. A statistical model validating triage for the peer review process: keeping the competitive applications in the review pipeline.
To determine what factors of the peer review process played a significant role in the assignment of scores for scientific merit by a major cardiovascular research finding agency. Specific variables studied included scores of internal and external reviewers, committee assignment and year of application.
                Retrospective analysis of research proposals submitted to the Heart and Stroke Foundation of Ontario from 1990 to 1994.
                Of a total database of 804 grant-in-aid (operating or project) proposals submitted to the Heart and Stroke Foundation of Ontario from 1990 to 1994, 779 had complete data and were included in the study. Variables analyzed were scores of internal and external reviewers, final committee score, committee assignment and year of application.
                There was poor correlation (r = 0.113) between external reviewers of grant proposals and only slightly better correlation between internal reviewers (r = 0.331). When mean internal and external scores were analyzed, correlation improved (r = 0.451) but agreement beyond chance was only moderate (weighed Cohen's kappa = 0.532). Regression analysis established that mean internal score had a greater impact on final committee score (r2 = 0.607, P < 0.001) than mean internal score (r2 = 0.348, P < 0.001). When review committee was entered as a dummy variable into a regression statement, it was statistically significant (P < 0.001) but explained less than 8% of variance in the final committee score. Mean scores showed a small 'upward creep' over the five-year period but explained less than 4% of variance in final committee score (r2 = 0.039, P < 0.001).
                In the peer review system studied, the scores of internal reviewers were more closely correlated to final committee score for scientific merit than those of external reviewers. Nevertheless, final committee scores were significantly different from either internal or external scores, suggesting that the process of committee discussion is an important, and highly influential, step in the peer review process. One hundred and thirteen grant applications were initially reviewed with only a brief outline of the applicant and the proposal and were later reviewed in more detail with the reports of outside experts. The detailed discussion and referees' reports had little impact on the original ratings. We compared the review patterns of applications for program projects and centers in allergy and immunology, reviewed by the National Institute of Allergy and Infectious Diseases review committee, with those of individual research projects in immunology, reviewed by three Division of Research Grants study sections. A high concordance between these review groups was obtained with respect to the application approval rate, mean score, and distribution of applications in various priority score ranges. Furthermore, we observed a high degree of consensus among reviewers on the quality of many program project and center applications, especially those with priority scores between 100 and 200. The National Institute of Health (NIH) peer review process for research grant applications is one of the largest and most respected systems of its kind in the world. Recently, however, the distribution of raw priority scores voted by NIH study sections has been skewed, and the rating behavior of individual review groups has been quite variable. These phenomena have made funding decisions more difficult. To achieve greater uniformity of rating behavior and a broader description of scores, an experiment was conducted involving 24 study sections. Standard adjectival descriptors and standard rating scales were used. On a random basis, half of the study sections were instructed to vote in units of 0.1 while the other half used an interval of 0.5. The results of this study have now been translated into standard practice at NIH. An experiment in which 150 proposals submitted to the National Science Foundation were evaluated independently by a new set of reviewers indicates that getting a research grant depends to a significant extent on chance. The degree of disagreement within the population of eligible reviewers is such that whether or not a proposal is funded depends in a large proportion of cases upon which reviewers happen to be selected for it. No evidence of systematic bias in the selection of NSF reviewers was found. nan Triage of grant application at the National Institutes of Health (NIH) is a process whereby an initial screening of applications by a scientific peer review group eliminates applications that are not competitive for awards. The process of application triage has been limited to those applications submitted to the NIH in response to an RFA (Request for Applications). A hypergeometric model was developed to determine the extent to which five, six, seven, or eight member triage teams or subsets of 12-to-20 member full committees could provide a statistically defensible triage decision. Although the intent of triage is to remove from review those applications that are noncompetitive, the model was weighted in favor of the applicant to minimize the likelihood that highly competitive applications would be eliminated. Within the assumptions and rules developed, it was determined that there was little likelihood that the latter would occur. For example, in the worst case scenario, the greatest probability that a highly competitive application would be knocked out of competition is P < or = 0.014 in the case of a five-member triage subset of a 20-member committee. Using the latter case, the model was tested on a set of 73 applications that were submitted to the National Cancer Institute for action at the February 1993 National Cancer Advisory Board. The model selected for triage required that each application be assigned to five reviewers, that each reviewer be blinded to the review assignments of the other reviewers, and that four noncompetitive votes be registered to triage out an application. Each of 19 applications received four to five noncompetitive votes, and were triaged out of the review process. The remaining 54 applications were then reviewed according to the usual NIH review process. Four of the applications received three noncompetitive triage votes each and were either rated as not recommended for further consideration (NRF, n = 2)) or received priority scores > or = 250 (n = 2) (The smaller the priority score the better the technical merit). Thirteen of the 53 applications received two noncompetitive votes. Of the latter, two were not recommended for further consideration and the remaining 11 received priority scores between in excess of 200. The distribution of competitive applications was such that funding was limited to those applications with priority scores of less than 190. Thus, the data suggest that the conservative model is valid such that the likelihood of eliminating a highly competitive application from consideration for funding is remotely small.(ABSTRACT TRUNCATED AT 400 WORDS)
There is little empirical evidence on the effects of grant giving peer review. No studies assessing the impact of peer review on the quality of funded research are presently available. Experimental studies assessing the effects of grant giving peer review on importance, relevance, usefulness, soundness of methods, soundness of ethics, completeness and accuracy of funded research are urgently needed. Practices aimed to control and evaluate the potentially negative effects of peer review should be implemented meanwhile.