Trial outcomes and statistical analyses
SECONDARY
The main outcome of this trial was the level of agreement on the top organizational items between the two study groups after round 3. This was assessed by examining the overlap between the most important items in each group and by calculating Krippendorff’s alpha coefficient on the aggregate rank order of items in the two groups. Krippendorff’s alpha is a reliability measure that has been proposed as the standard reliability statistic due to its flexibility and advantages over other known reliability coefficients [The secondary outcomes were as follow: (1) time to complete round two and round three questionnaires, measured separately, and (2) self-reported ease or difficulty with the assessment task, measured by the Single Ease Question (SEQ) administered immediately after the final assessment of items. The SEQ is a standard user metric in the form of a 7-point rating scale (from 1 “very difficult” to 7 “very easy”) that is simple, quick, and has proven to perform as well or better than more complicated measures of task difficulty [