BackgroundAlthough autophagy is an important mediator of metformin antitumor activity, the role of metformin in the crosstalk between autophagy and apoptosis remains unclear. The aim was to confirm the anticancer effect by inducing apoptosis by co-treatment with metformin and OSMI-1, an inhibitor ofO-GlcNAcylation, in colon cancer cells.MethodsCell viability was measured by MTT in colon cancer cell lines HCT116 and SW620 cells. Co-treatment with metformin and OSMI-1 induced autophagy and apoptosis, which was analyzed using western blot, reverse transcription-polymerase chain reaction (RT-PCR) analysis, and fluorescence-activated cell sorting (FACS). Combined treatment with metformin and OSMI-1 synergistically inhibit the growth of HCT116 was confirmed by xenograft tumors.ResultsWe showed that metformin inhibited mammalian target of rapamycin (mTOR) activity by inducing high levels of C/EBP homologous protein (CHOP) expression through endoplasmic reticulum (ER) stress and activating adenosine monophosphate-activated protein kinase (AMPK) to induce autophagy in HCT116 cells. Interestingly, metformin increasedO-GlcNAcylation and glutamine:fructose-6-phosphate amidotransferase (GFAT) levels in HCT116 cells. Thus, metformin also blocks autophagy by enhancingO-GlcNAcylation, whereas OSMI-1 increases autophagy via ER stress. In contrast, combined metformin and OSMI-1 treatment resulted in continuous induction of autophagy and disruption ofO-GlcNAcylation homeostasis, resulting in excessive autophagic flux, which synergistically induced apoptosis. Downregulation of Bcl2 promoted apoptosis via the activation of c-Jun N-terminal kinase (JNK) and CHOP overexpression, synergistically inducing apoptosis. The activation of IRE1α/JNK signaling by OSMI-1 and PERK/CHOP signaling by metformin combined to inhibit Bcl2 activity, ultimately leading to the upregulation of cytochrome c release and activation of caspase-3.ConclusionsIn conclusion, combinatorial treatment of HCT116 cells with metformin and OSMI-1 resulted in more synergistic apoptosis being induced by enhancement of signal activation through ER stress-induced signaling rather than the cell protective autophagy function. These results in HCT116 cells were also confirmed in xenograft models, suggesting that this combination strategy could be utilized for colon cancer treatment.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12935-023-02954-2.