Discussion
In this diagnostic study, we examined the novel use of deep-learning–based NLP to measure a complex outcome from unstructured EHR text in a large pragmatic clinical trial. We also demonstrated and validated the use of statistical methods to quantitatively assess the effects of NLP-related misclassification on study power at a given sample size.Natural language processing–screened human abstraction represents an efficient and useful approach for measuring EHR outcomes in large pragmatic studies and is increasingly used to measure palliative care outcomes similar to the one we examined.Researchers considering NLP or NLP-aided approaches to measuring outcomes should consider the effects of the outcome measurement strategy on the statistical power, costs, and validity of the trial. In this study, we demonstrated 2 simple methods for researchers to perform misclassification-adjusted power calculations, and we encourage the uptake of this approach in the design of future trials.Perhaps the most conspicuous limitation of using NLP to measure clinical research outcomes is the investment required to implement NLP. Although this investment may be minimal for outcomes that are easily detected using rule-based NLP, identifying more complex constructs such as the one investigated here may require substantial software development and acquisition of training and validation data. Our research group has spent more than 500 developer-hours implementing this NLP model, in addition to 491 abstractor-hours spent collecting training and validation data. We anticipate that the cost of implementing high-performing pretrained NLP models will decrease as the field matures. Additionally, many NLP development costs are fixed with respect to the number of study participants, and many aspects of NLP development are transferrable to other studies and research questions. Although our BERT NLP model is not yet portable due to the privacy risks associated with training on identifiable protected health information,