Iteration effect
In the Multi-Round dataset, the variance in ratings reduced between rounds indicating an ‘iteration effect’ (i.e., the converging of scores). In the RTD dataset, most outcome ratings did not change over time, so the variance overall did not reduce, indicating no ‘iteration effect’ overall. However, in those outcomes that were re-rated in the RTD dataset, the variance was reduced substantially. This suggests that most experts in the RTD were happy with their initial ratings, and where they were not, they amended them substantially. There did not seem to be much advantage in participants having to re-rate their outcomes over subsequent rounds, as convergence in the RTD dataset, where re-ratings were allowed but not enforced, may be better.At first glance, the ‘feedback’ and ‘iteration effect’ results seem slightly contradictory. The slope of the ‘feedback effect’ (Fig. 