In order to develop new anti-cancer drugs more efficiently and reduce side effects based on active drug targets, the virtual drug screening was carried out through the target of G-quadruplexes and 23 hit compounds were, thus, screened out as potential anticancer drugs. Six classical G-quadruplex complexes were introduced as query molecules, and the three-dimensional similarity of molecules was calculated by shape feature similarity (SHAFTS) method so as to reduce the range of potential compounds. Afterwards, the molecular docking technology was utilized to perform the final screening followed by the exploration of the binding between each compound and four different structures of G-quadruplex. In order to verify the anticancer activity of the selected compounds, compounds 1, 6 and 7 were chosen to treat A549 cells in vitro, the lung cancer epithelial cells, for further exploring their anticancer activity. These three compounds were found to be of good characteristics in the treatment of cancer, which revealed the great application prospect of the virtual screening method in developing new drugs.