Strengths and limitations
Our study has several strengths. It is the first multi-site, randomized study of problem list alerts done using a variety of EHRs and had a relatively effective intervention. It also had some important limitations. First, we did not assess whether the problems added from the alerts were accurate—we believe that they largely were, but there may have been some false positives. Second, although we had 4 sites, there were many differences in implementation strategy at each site, which meant we could not assess whether specific intervention characteristics (eg, inpatient vs outpatient, interruptive vs noninterruptive, and actionable vs nonactionable) made a difference. More sites or a longer time period of data collection, and a random allocation of alert features would be needed to draw these conclusions. Third, our data sharing plan did not include the sharing of additional baseline data about the rate of problem list utilization or number of encounters at each site, so we could not make direct comparisons beyond alert volume, acceptance, and the number of problems added at each site. However, since this study was a randomized controlled trial, we expect these to be well-balanced across arms at each site. Fourth, we focused, in this study, only on adding missing problems and not on removing inaccurate, unimportant, or redundant problems—problem lists frequently contain inaccurate or out-of-date problem entries, so this would be a useful topic for future research. Fifth, our study looked only at structured data already in the medical record to infer problems—by using, for example, natural language processing on notes, optical character recognition of scanned documents, and integration of external data (eg, through a health information exchange) additional problems could potentially be identified, possibly with higher specificity. Finally, more fully automated strategies, such as automatic creation and maintenance of the problem list could also be explored with a goal of reducing the overhead of maintaining the problem list for clinicians—such a strategy would need to be weighed against possible issues with accuracy, as well as the possibility that clinician curation of the problem list may have benefits as the clinician thinks through a patient’s problems.