Viruses are obligate intracellular parasites that critically depend on their hosts to initiate infection, complete replication cycles, and generate new progeny virions. To achieve these goals, viruses have evolved numerous elegant strategies to subvert and utilize different cellular machinery. The cytoskeleton is often one of the first components to be hijacked as it provides a convenient transport system for viruses to enter the cell and reach the site of replication. The cytoskeleton is an intricate network involved in controlling the cell shape, cargo transport, signal transduction, and cell division. The host cytoskeleton has complex interactions with viruses during the viral life cycle, as well as cell-to-cell transmission once the life cycle is completed. Additionally, the host also develops unique, cytoskeleton-mediated antiviral innate immune responses. These processes are also involved in pathological damages, although the comprehensive mechanisms remain elusive. In this review, we briefly summarize the functions of some prominent viruses in inducing or hijacking cytoskeletal structures and the related antiviral responses in order to provide new insights into the crosstalk between the cytoskeleton and viruses, which may contribute to the design of novel antivirals targeting the cytoskeleton.