Main findings
In this randomized trial, we found moderately high reliability between the prioritization of top items in a rating arm and a ranking arm at the end of a Delphi process (Krippendorff alpha = 0.811, 95% CI = 0.669–0.920). Krippendorff states that it is customary to require alpha ≥ 0.8 to ensure that the data under consideration are similarly interpretable by different coders and that ≥ 0.667 is the lower limit where tentative conclusions are acceptable [Contrary to common practice in Delphi, we did not define a consensus threshold to compare the final outputs of our experimental groups. Rather, we viewed the prioritization of top items as providing a fairer comparison between rating and ranking. Had we done so, it is likely that the final set of items in the ranking group would have included fewer items than in the rating group, as ranking prohibits ties. This is apparent in the results of the third round (Additional file This study also found that the ranking task was significantly more difficult. The 9-min mean difference to complete the third-round questionnaire in favor of the rating group failed to achieve statistical significance (