BackgroundCancer poses a health threat, with an increased incidence worldwide. Thus, it is essential to develop new natural anticancer agents.Dypsis pembana(H.E.Moore) Beentje & J.Dransf (DP) is an ornamental plant belonging to the family Arecaceae. This study aimed to isolate and identify phytoconstituents from the leaves of this plant and evaluate their in vitro cytotoxic activities.MethodsDifferent chromatographic techniques were applied to fractionate the hydro-alcoholic extract of DP and separate the major phytoconstituents. The isolated compounds were structurally elucidated based on their physical and spectroscopic data. The in vitro cytotoxic activities of the crude extract and fractions thereof were evaluated against human colon carcinoma (HCT-116), human breast carcinoma (MCF-7), and human hepatocellular carcinoma (HepG-2) cell lines via MTT assay. Moreover, selected isolates were tested against HepG-2 cell line. Molecular docking analysis was performed to investigate the interactions of these compounds with two potential targets, the human topoisomerase IIα and cyclin-dependent kinase 2 enzymes.ResultsThirteen diverse compounds were reported for the first time from DP, providing significant chemotaxonomic biomarkers. Among tested compounds, vicenin-II (7) was the most cytotoxic against HepG-2 cell line, with an IC50value of 14.38 µg/mL, followed by isovitexin (13) (IC50of 15.39 µg/mL). These experimental findings were complemented by molecular docking, which demonstrated that vicenin-II exhibited superior enzyme-binding affinities to the studied vital targets and shed light on the structure–activity relationships among the investigated flavone-C-glycosides members.ConclusionThe phytochemical profile of DP was characterized for the first time, reflecting chemotaxonomic data about the concerned species, genus, or even the family. Biological and computational findings revealed that vicenin-II and isovitexin are possible lead structures as inhibitors of the human topoisomerase IIα and cyclin-dependent kinase 2 enzymes.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12906-023-04046-0.