BackgroundDepression is a neuropsychiatric disease resulting from deteriorations of molecular networks and synaptic injury induced by stress. Traditional Chinese formula Xiaoyaosan (XYS) exert antidepressant effect, which was demonstrated by a great many of clinical and basic investigation. However, the exact mechanism of XYS has not yet been fully elucidated.MethodsIn this study, chronic unpredictable mild stress (CUMS) rats were used as a model of depression. Behavioral test and HE staining were used to detect the anti-depressant effects of XYS. Furthermore, whole transcriptome sequencing was employed to establish the microRNA (miRNA), long non-coding RNA (lncRNA), circular RNA (circRNA), and mRNA profiles. The biological functions and potential mechanisms of XYS for depression were gathered from the GO and KEGG pathway. Then, constructed the competing endogenous RNA (ceRNA) networks to illustrate the regulatory relationship between non-coding RNA (ncRNA) and mRNA. Additionally, longest dendrite length, total length of dendrites, number of intersections, and density of dendritic spines were detected by Golgi staining. MAP2, PSD-95, SYN were detected by immunofluorescence respectively. BDNF, TrkB, p-TrkB, PI3K, Akt, p-Akt were measured by Western Blotting.ResultsThe results showed that XYS could increase the locomotor activity and sugar preference, decreased swimming immobility time as well as attenuate hippocampal pathological damage. A total of 753 differentially expressed lncRNAs (DElncRNAs), 28 circRNAs (DEcircRNAs), 101 miRNAs (DEmiRNAs), and 477 mRNAs (DEmRNAs) were identified after the treatment of XYS in whole transcriptome sequencing analysis. Enrichment results revealed that XYS could regulate multiple aspects of depression through different synapse or synaptic associated signal, such as neurotrophin signaling and PI3K/Akt signaling pathways. Then, vivo experiments indicated that XYS could promote length, density, intersections of synapses and also increase the expression of MAP2 in hippocampal CA1, CA3 regions. Meanwhile, XYS could increase the expression of PSD-95, SYN in the CA1, CA3 regions of hippocampal by regulating the BDNF/trkB/PI3K signal axis.ConclusionThe possible mechanism on synapse of XYS in depression was successfully predicted. BDNF/trkB/PI3K signal axis were the potential mechanism of XYS on synapse loss for its antidepressant. Collectively, our results provided novel information about the molecular basis of XYS in treating depression.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12906-023-04000-0.