BackgroundMedication adherence is usually defined as the extent of the agreement between the medication regimen agreed to by patients with their healthcare provider and the real-world implementation. Proactive identification of those with poor adherence may be useful to identify those with poor disease control and offers the opportunity for ameliorative action. Adherence can be estimated from Electronic Health Records (EHRs) by comparing medication dispensing records to the prescribed regimen. Several methods have been developed in the literature to infer adherence from EHRs, however there is no clear consensus on what should be considered the gold standard in each use case.Our objectives were to critically evaluate different measures of medication adherence in a large longitudinal Scottish EHR dataset. We used asthma, a chronic condition with high prevalence and high rates of non-adherence, as a case study.MethodsOver 1.6 million asthma controllers were prescribed for our cohort of 91,334 individuals, between January 2009 and March 2017. Eight adherence measures were calculated, and different approaches to estimating the amount of medication supply available at any time were compared.ResultsEstimates from different measures of adherence varied substantially. Three of the main drivers of the differences between adherence measures were the expected duration (if taken as in accordance with the dose directions), whether there was overlapping supply between prescriptions, and whether treatment had been discontinued. However, there are also wider, study-related, factors which are crucial to consider when comparing the adherence measures.ConclusionsWe evaluated the limitations of various medication adherence measures, and highlight key considerations about the underlying data, condition, and population to guide researchers choose appropriate adherence measures. This guidance will enable researchers to make more informed decisions about the methodology they employ, ensuring that adherence is captured in the most meaningful way for their particular application needs.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12874-023-01935-3.