Lessons learned
TRANSFORMATION
The first lesson learned from this project is particularly striking — only 1 in 8 US sites participating in a large cardiovascular outcomes trial were interested in or able to participate in this effort to demonstrate how to ascertain data from EHR. Even when offered a more flexible set of options for data ascertainment, many interested sites were unable to participate in the demonstration project. A surprisingly high proportion (25%) did not have an EHR; among those with an EHR, many were either simply not interested (64.8%) or were non-responsive to the feasibility survey (9.3%), despite repeated email outreach to all institutions and telephone calls to the investigators at sites that were affiliated with a hospital system. The investigators at several sites were very interested in participating but were unable to get buy-in from the IT and/or contracting teams at their institutions. Engagement attempts were made in late 2019/early 2020 so response rates were likely affected by the strain of the COVID-19 pandemic. Although our study was a demonstration project and may not be fully representative of sites that may participate in future EHR-sourced trials, the widespread lack of site interest or ability to participate in EHR-based efforts has important implications for the scalability of such models. Given the substantial interest in using EHR within clinical trials across academic medical centers, governmental organizations, and industry sponsors, there is a need to further understand the landscape of site perspectives, capabilities, and performance, including site-level barriers to participation. Future EHR-sourced trials should consider the potential impact of EHR-related site selection on the generalizability of study findings, particularly if patient characteristics are substantially different from those of sites who are able to participate.Second, our experiences with site selection and start-up paralleled those in many conventional clinical trials. One participating site did not enroll any patients in the demonstration project, in part due to contracting delays which led to few patient visits prior to the final query date. Similar to many prospective studies, there was variation in site performance with respect to study enrollment, with some sites enrolling much more effectively than others. In general, Central Transformation sites enrolled higher numbers of patients in the trial and had better demonstration project enrollment yield than Network Consortium sites. Given that the demonstration project largely overlapped with the first year of the COVID-19 pandemic, it is quite possible that enrollment in the trial and the demonstration project were paused due to the local diversion of research staff or the reallocation of resources. This problem may have disproportionally affected the Network Consortium sites, which are generally part of large academic health systems where the early COVID-19 burden was high.Third, we observed important variations in regulatory and contracting capabilities by site. At one large academic center which used the Network Consortium strategy, EHR data are owned by the health system, but trial operations are generally executed by a separate entity. As a result, this site required separate contracts for these two entities, which contributed to a longer start-up process. Most Central Transformation sites used an external IRB, which may have supported more rapid completion of start-up activities, but this pattern was not universal; one Central Transformation site did not have its own IRB and did not have sufficient experience working with a central IRB for study oversight. This site agreed to rely on Duke’s IRB and a coordinating center team member drafted the application for submission and handled the IRB submissions. Given the novelty of the EHR-sourced approach for many trial sites, central availability of regulatory expertise and contracting resources are critical for ensuring that sites with varying levels of experience can be supported through the startup process.Finally, even within the pre-defined data collection strategy types, we observed variations in the technical capabilities and ultimate approaches to data submission. The process for adapting the broad strategies at each site was iterative, with site-specific processes refined based on local data structure and central review of preliminary data submissions. All of the Central Transformation sites had to make corrections to their preliminary submissions. There were no resubmissions from the Network Consortium sites, likely due to our use of the same data domains previously used for other studies, which have each been through prior quality checks. At one Central Transformation site, the site did not receive an automated extract of inpatient data from the local hospital. As a result, site staff had to manually extract the data from the hospital EHR. From a coordinating center perspective, the quality control effort to evaluate data from this site was not substantially higher than for other Central Transformation sites; however, this may not be the case for trials with a large number of participants and/or many variables. Given the manual nature of the site-level work, however, the total effort required was much higher than for other Central Transformation sites. The technical lessons learned for Central Transformation sites are twofold: (1) providing central support to sites for data extraction was critical for enabling the capture of all key data elements; and (2) this support was continuous, relying on multiple rounds of review rather than a single orientation at study onset.These lessons learned reflect a single trial experience, which may not be directly applicable to different types of trials or in settings/countries with heterogeneous data sources and trial processes. Additionally, though we included quantitative findings, any comparisons should be taken as preliminary and hypothesis generating. However, these observations are valuable when synthesized into the following recommendations.