Methods
chronic life-limiting illness, ADRD, Alzheimer disease, dementias
ALZHEIMER DISEASE
This diagnostic study was conducted to inform selection of an outcome measurement strategy for a pragmatic randomized clinical trial of a communication-priming intervention for hospitalized patients (Project to Improve Communication About Serious Illness—Hospital Study: Pragmatic Trial 1 [PICSI-H Trial 1]).The trial enrolled patients hospitalized at any of 3 study hospitals who either had advanced age (≥80 years) or were aged 55 years or older and had a chronic life-limiting illness as defined by diagnosis codes used by the Dartmouth Atlas Project to study end-of-life care (eTable 1 in During planning, the trial was specified to use NLP to measure its primary outcome, with human abstraction as a backup strategy. However, because NLP approaches were developed concurrently with enrollment, the expected degree of NLP-related misclassification was not known at the time the trial began. The trial was initially specified to target a sample size of 2000 participants, which would result in 80% power to detect a difference in proportions of at least 6.2% (assuming a control arm proportion of 0.54 and 2-sided α of 0.05). However, this sample size determination did not consider the potential effect of NLP-related misclassification. Development, training, and testing of NLP models continued throughout enrollment, using data sources gathered from outside the trial.Between April 23, 2020, and March 26, 2021, the trial enrolled 2512 patients. The prespecified enrollment target of 2000 was exceeded to increase the number of participants with Alzheimer disease and related dementias (ADRD), a prespecified subgroup. Following conclusion of enrollment and prior to unblinding and primary analyses, we froze our NLP program, evaluated NLP performance within a validation sample collected from the trial, and reevaluated the statistical power, human abstraction burden, and pragmatic implications of 3 strategies for measuring the primary outcome: (1) conventional manual abstraction, (2) NLP alone, and (3) NLP-screened human abstraction, in which only EHR passages scored by NLP above a predefined threshold would be reviewed by human abstractors for documented goals-of-care discussions.