BackgroundMicroRNAs are a group of small non-coding RNAs that are involved in development and diseases such as cancer. Previously, we demonstrated that miR-335 is crucial for preventing collagen type XI alpha 1 (COL11A1)-mediated epithelial ovarian cancer (EOC) progression and chemoresistance. Here, we examined the role of miR-509-3p in EOC.MethodsThe patients with EOC who underwent primary cytoreductive surgery and postoperative platinum-based chemotherapy were recruited. Their clinic-pathologic characteristics were collected, and disease-related survivals were determined. The COL11A1 and miR-509-3p mRNA expression levels of 161 ovarian tumors were determined by real-time reverse transcription-polymerase chain reaction. Additionally, miR-509-3p hypermethylation was evaluated by sequencing in these tumors. The A2780CP70 and OVCAR-8 cells transfected with miR-509-3p mimic, while the A2780 and OVCAR-3 cells transfected with miR-509-3p inhibitor. The A2780CP70 cells transfected with a small interference RNA of COL11A1, and the A2780 cells transfected with a COL11A1 expression plasmid. Site-directed mutagenesis, luciferase, and chromatin immunoprecipitation assays were performed in this study.ResultsLow miR-509-3p levels were correlated with disease progression, a poor survival, and high COL11A1 expression levels. In vivo studies reinforced these findings and indicated that the occurrence of invasive EOC cell phenotypes and resistance to cisplatin are decreased by miR-509-3p. The miR-509-3p promoter region (p278) is important for miR-509-3p transcription regulation via methylation. The miR-509-3p hypermethylation frequency was significantly higher in EOC tumors with a low miR-509-3p expression than in those with a high miR-509-3p expression. The patients with miR-509-3p hypermethylation had a significantly shorter overall survival (OS) than those without miR-509-3p hypermethylation. Mechanistic studies further indicated that miR-509-3p transcription was downregulated by COL11A1 through a DNA methyltransferase 1 (DNMT1) stability increase. Moreover, miR-509-3p targets small ubiquitin-like modifier (SUMO)-3 to regulate EOC cell growth, invasiveness, and chemosensitivity.ConclusionThe miR-509-3p/DNMT1/SUMO-3 axis may be an ovarian cancer treatment target.Supplementary InformationThe online version contains supplementary material available at 10.1186/s13048-023-01191-5.