BackgroundTaohong Siwu Decoction (THSWD) is a widely used traditional Chinese medicine (TCM) prescription in the treatment of ischemic stroke. There are thousands of chemical components in THSWD. However, the key functional components are still poorly understood. This study aimed to construct a mathematical model for screening of active ingredients in TCM prescriptions and apply it to THSWD on ischemic stroke.MethodsBotanical drugs and compounds in THSWD were acquired from multiple public TCM databases. All compounds were initially screened by ADMET properties. SEA, HitPick, and Swiss Target Prediction were used for target prediction of the filtered compounds. Ischemic stroke pathological genes were acquired from the DisGeNet database. The compound–target–pathogenic gene (C-T-P) network of THSWD was constructed and then optimized using the multiobjective optimization (MOO) algorithm. We calculated the cumulative target coverage score of each compound and screened the top compounds with 90% coverage. Finally, verification of the neuroprotective effect of these compounds was performed with the oxygen-glucose deprivation and reoxygenation (OGD/R) model.ResultsThe optimized C-T-P network contains 167 compounds, 1,467 predicted targets, and 1,758 stroke pathological genes. And the MOO model showed better optimization performance than the degree model, closeness model, and betweenness model. Then, we calculated the cumulative target coverage score of the above compounds, and the cumulative effect of 39 compounds on pathogenic genes reached 90% of all compounds. Furthermore, the experimental results showed that decanoic acid, butylphthalide, chrysophanol, and sinapic acid significantly increased cell viability. Finally, the docking results showed the binding modes of these four compounds and their target proteins.ConclusionThis study provides a methodological reference for the screening of potential therapeutic compounds of TCM. In addition, decanoic acid and sinapic acid screened from THSWD were found having potential neuroprotective effects first and verified with cell experiments, however, further in vitro and in vivo studies are needed to explore the precise mechanisms involved.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12906-023-03990-1.