Evaluating NLP Performance
ADRD
Following model training and manual abstraction of the validation sample, we used the trained BERT NLP model to predict the probability of documented goals-of-care discussions in all 2.64 million EHR passages (44 324 notes) of the PICSI-H Trial 1 data set. To characterize the expected performance of BERT NLP in the trial data set, we resampled the validation sample to reflect the prevalence of enrolled patients with ADRD (11%) and compared NLP-predicted probabilities for each note and patient against manual abstraction using ROC curves and precision-recall analyses. For all analyses, we report note- and patient-level performance, defining note- and patient-level probability as the maximum predicted probability for all constituent passages and defining the gold-standard label for each note and patient as the union of labels for all constituent passages. Statistical analyses were conducted using Stata/MP, version 17.0 (StataCorp LLC)