Comparison of Outcome Measurement Strategies
fatigue
We evaluated 3 strategies for measuring the primary outcome: manual human abstraction, BERT NLP alone, and BERT NLP–screened human abstraction. The first approach, manual human abstraction, is the de facto gold standard and would power the study to detect a risk difference of 5.4%. Our experience suggested that individual abstractors could perform this abstraction task for up to 3 hours per day before experiencing excessive fatigue. To estimate the number of hours required for complete manual abstraction of the trial data set, we scaled the known abstractor-hours required to collect data for the validation sample to the entire trial data set, yielding an estimate of approximately 3000 abstractor-hours (ie, 67 work weeks for a team of 3 abstractors devoting 3 hours per day to this task; costing $195 000 at a rate of $65 per hour). Based on estimates from the validation sample, constraining abstractors to records from randomization to the first EHR-documented goals-of-care discussion (or 30 days if none was present) would reduce this estimate to approximately 2000 abstractor-hours (ie, 45 work weeks for a team of 3 abstractors, or $130 000 at a rate of $65 per hour).In the second approach, BERT NLP alone, the primary outcome could be measured with any combination of sensitivity and specificity represented on the ROC curve in In the third approach, BERT NLP–screened human abstraction, only EHR passages that were scored by NLP above a predefined threshold would be reviewed by human abstractors for documented goals-of-care discussions.