BackgroundInsulin resistance (IR) is an important determinant of glucose metabolic disturbance and placental dysplasia in gestational diabetes mellitus (GDM). Calcium/calmodulin dependent protein kinase IV (CAMK4) improves insulin IR induced by a high-fat diet (HFD). The current study sought to elucidate the role and potential mechanism of CAMK4 in GDM.MethodsA GDM model was established in female C57BL/6J mice via HFD feeding for one week before mating and throughout gestation. The IR was elicited by 10–6M insulin treatment for 48 h in HTR-8/SVneo cells and mouse primary trophoblast cells. The function of CAMK4 was investigated by transfection of overexpression plasmid in HTR-8/SVneo cells and infection of lentivirus loaded with CAMK4 encoding sequence in primary trophoblast cells. Real-time PCR, western blot, cell counting kit-8, transwell, wound healing, dual-luciferase reporter assay, and liquid chromatography/mass spectrometry-based untargeted metabolomics were performed to confirm the effects of CAMK4 on trophoblast cells.ResultsDecreased CAMK4 expression was found in the placenta of GDM mice. CAMK4 overexpression ameliorated IR-induced viability impairment, migratory and invasive capacity inhibition, autophagy blocking, insulin signaling inactivation and glucose uptake disorder in trophoblast cells. CAMK4 also transcriptionally activated orphan nuclear receptor NUR77, and the effects of CAMK4 were abrogated by silencing of NUR77. Metabolomics analysis revealed that CAMK4 overexpression caused alterations of amino acid, lipid and carbohydrate metabolism, which were important in GDM.ConclusionOur results indicated that CAMK4/NUR77 axis may provide novel potential targets in GDM treatment.Supplementary InformationThe online version contains supplementary material available at 10.1186/s10020-023-00669-8.