of individual publications rather than at some aggregate level. This is an important difference that we revisit in Section \[sec:level\_of\_aggregation\].

The REF 2014 results were also analysed by @Mryglod2015 [@Mryglod2015a] at the institutional level. They found that the departmental $h$-index was not sufficiently predictive, even though an earlier analysis suggested that the $h$-index might be predictive in Psychology [@Bishop2014]. An analysis by Elsevier found that metrics were reasonably predictive of peer review outcomes at an institutional level in some fields but not in others [@Jump2015].

Both @Pride and @Harzing2017 compared the UK REF results with metrics using Microsoft Academic Graph [@Harzing2017a]. @Pride compared the median number of citations with the REF GPA, which is a weighted average of the proportion of publications that have been awarded a certain number of stars for all UoAs, clearly taking a size-independent perspective. They found correlations on the order of $0.7$–$0.8$ for the UoAs that showed the highest correlations. @Harzing2017 compared the total number of citations and a so-called REF power rating, taking a size-dependent perspective, and found a very high correlation of $0.97$. This correlation was obtained at an even higher aggregate level, namely, the overall institutional level, without differentiating between different disciplines. 