PurposeAs the global population ages rapidly, osteoporotic fractures have become an important public health problem. Previous studies have suggested that miR-137 is involved in the regulation of bone formation, but its specific regulatory mechanism remains unclear. In this study, we aimed to explore the expression, role, and regulatory mechanism of miR-137 in the osteogenic differentiation of human bone marrow mesenchymal stem cells (hBMSCs).MethodshBMSCs were induced into osteoblasts at first, and the expression level of miR-137 at different time points was detected. After knockdown and overexpression of miR-137, the effect of miR-137 on the osteogenic differentiation of hBMSCs was examined through alkaline phosphatase (ALP) staining and Alizarin Red staining. Western blotting was performed to detect the expression of runt-related transcription factor 2 (Runx2), osteocalcin (OCN), and toll-like receptor 4 (TLR4)/nuclear factor-κB (NF-κB) pathway. Bioinformatics websites were used to predict the target binding sites for miR-137 and KDM4A, and the results were validated using luciferase reporter gene experiments. Moreover, the ALP activity, calcium nodule formation, and activation of Runx2, OCN, and TLR4/NF-κB pathways were observed after knockdown of KDM4A.ResultsThe expression of miR-137 decreased during osteogenic differentiation. Knockdown of miR-137 expression increased the osteogenic ability of hBMSCs, while overexpression of it weakened the ability. Through the activation of the TLR4/NF-κB pathway, miR-137 inhibited osteogenic differentiation. KDM4A was identified as a predicted target gene of miR-137. After knocking down KDM4A expression, the osteogenic ability of hBMSCs was diminished, and the TLR4/NF-κB pathway was activated. Furthermore, the osteogenic ability of hBMSCs was partially restored and the activation level of TLR4/NF-κB was reduced after miR-137 knockdown.ConclusionMiR-137 enhances the activity of the TLR4/NF-κB pathway by targeting KDM4A, thereby inhibiting the osteogenic differentiation of hBMSCs and exacerbating osteoporosis.Supplementary InformationThe online version contains supplementary material available at 10.1186/s13018-023-03918-y.