world data of over 63,000 peer grades. Their models are probabilistic so give a belief distribution over grades (as opposed to a single score) for each student.

PEAS is a new peer evaluation extension for the EdX open source MOOC platform [@peas]. Students are incentivized to grade accurately by a calibration method that constructs an incentive score based on the accuracy of their grading. To improve review quality, students are divided into groups based on this incentive score, and each assignment is peer graded by one student from each group. A simple normalisation of grades is also performed to reduce bias in peer grading. Expert grading can be used to resolve discrepancies in peer grades, and to provide training data for Machine Learning algorithms to grade automatically. An important difference with our work is that the calibration in PEAS is just once, whilst PeerRank potentially uses multiple rounds of adjustment of grades.

One of the closest works to ours is a peer reviewing mechanism being piloted by the National Science Foundation (NSF) for the Signal and Sensing Systems (SSS) program. The mechanism is designed to help deal with an increase in proposals which is putting an increasing stress on the 