BackgroundNeonatal hyperbilirubinemia (NH) is a major cause of hospitalization after birth. Previous studies indicated that vitamin D deficiency might play an important role in NH susceptibility, but the results were controversial. Meanwhile, there has been limited description of the association between vitamin D related genes single nucleotide polymorphisms (SNP) and NH susceptibility. We aimed to investigate the vitamin D metabolic pathway genes polymorphisms and vitamin D levels with NH susceptibility.MethodsWe retrospectively analyzed the clinical data, vitamin D levels and its metabolic pathway gene polymorphisms of 187 NH neonates and 149 controls at Tianjin Children’s Hospital/Tianjin University Children’s Hospital between April 2019 and August 2022. Vitamin D levels were measured by liquid chromatography-tandem mass spectrometry (LC-MS/MS) method, and the genetic polymorphism ofNADSYN1/DHCR7,GC,CYP2R1,CYP24A1andCYP27B1was detected by high resolution melting (HRM) analysis.ResultsThe frequency of vitamin D deficiency (25(OH)D < 15 ng/mL) was significantly increased in the NH group compared to controls. TT genotype of rs12785878 and GT genotype of rs10877012 were protective factors of vitamin D deficiency and NH, and GT genotype and dominant model carriers of rs12785878 had a higher risk of severe NH than the GG genotype carriers (GT genotype: OR: 2.43; 95% CI: 1.22–4.86;P= 0.012, dominant model: OR: 1.97; 95% CI: 1.04–3.73;P= 0.037).GCgene haplotype was associated with vitamin D deficiency. No significant SNP-SNP and SNP-vitamin D levels interaction combinations were found.ConclusionsThere were associations among NH, vitamin D deficiency andNADSYN1/DHCR7andCYP27B1polymorphisms, TT genotype of rs12785878 and GT genotype of rs10877012 could reduce the risk of vitamin D deficiency and NH. Furthermore, rs12785878 was significantly associated with severe NH.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12887-023-04086-y.