BackgroundThe Dutch population-based cervical cancer screening programme (PBS) consists of primary high-risk human papilloma virus (hrHPV) testing with cytology as triage test. In addition to cervical scraping by a general practitioner (GP), women are offered self-sampling to increase participation. Because cytological examination on self-sampled material is not feasible, collection of cervical samples from hrHPV-positive women by a GP is required. This study aims to design a methylation marker panel to detect CIN3 or worse (CIN3+) in hrHPV-positive self-samples from the Dutch PBS as an alternative triage test for cytology.MethodsFifteen individual host DNA methylation markers with high sensitivity and specificity for CIN3+ were selected from literature and analysed using quantitative methylation-specific PCR (QMSP) on DNA from hrHPV-positive self-samples from 208 women with CIN2 or less (< CIN2) and 96 women with CIN3+. Diagnostic performance was determined by area under the curve (AUC) of receiver operating characteristic (ROC) analysis. Self-samples were divided into a train and test set. Hierarchical clustering analysis to identify input methylation markers, followed by model-based recursive partitioning and robustness analysis to construct a predictive model, was applied to design the best marker panel.ResultsQMSP analysis of the 15 individual methylation markers showed discriminative DNA methylation levels between < CIN2 and CIN3+ for all markers (p< 0.05). The diagnostic performance analysis for CIN3+ showed an AUC of ≥ 0.7 (p< 0.001) for nine markers. Hierarchical clustering analysis resulted in seven clusters with methylation markers with similar methylation patterns (Spearman correlation> 0.5). Decision tree modeling revealed the best and most robust panel to contain ANKRD18CP, LHX8 and EPB41L3 with an AUC of 0.83 in the training set and 0.84 in the test set. Sensitivity to detect CIN3+ was 82% in the training set and 84% in the test set, with a specificity of 74% and 71%, respectively. Furthermore, all cancer cases (n= 5) were identified.ConclusionThe combination of ANKRD18CP, LHX8 and EPB41L3 revealed good diagnostic performance in real-life self-sampled material. This panel shows clinical applicability to replace cytology in women using self-sampling in the Dutch PBS programme and avoids the extra GP visit after a hrHPV-positive self-sampling test.Supplementary InformationThe online version contains supplementary material available at 10.1186/s13148-023-01517-6.