$4^*$ publications per top $10\%$ publication. This suggests that what is considered as $4^*$ publication in peer review differs per field, where some fields seem to use more stringent conditions than others. Similarly, @Wooding2015 found that peer review was less stringent in REF 2014 than in REF 2008, in what publications were considered worthy of $4^*$.

Before presenting our results, we first address four important methodological considerations. We start by reflecting on the level of aggregation at which agreement between metrics and peer review should be analysed. We then examine both the size-dependent and size-independent perspectives, especially regarding correlations. This leads us to consider alternative measures of agreement. Finally, we discuss the matter of peer review uncertainty.

Level of aggregation {#sec:level_of_aggregation}
--------------------

The *Metric Tide* report analysed agreement between metrics and peer review at the level of individual publications. We believe that this is not appropriate in the context of the REF, and it may explain the large differences between the *Metric Tide* report and preceding publications in which agreement between metrics and peer review was analysed. The institutional level is the appropriate level to use for the analysis. The analysis at the level of individual publications is very interesting. 