CD8+T cells are essential lymphocytes with cytotoxic properties for antitumor immunotherapy. However, during chronic infection or tumorigenesis, these cells often become dysfunctional with a gradually depleted ability to release cytokines and the exhibition of reduced cytotoxicity, the state referred to as “T-cell exhaustion” (Tex). This unique state was characterized by the increasing expression of inhibitory checkpoint receptors, and interventions targeting immune checkpoint blockades (ICBs) have been considered as a promising strategy to stimulate T-cell killing. Recent investigations have demonstrated that exhausted T cells not only display functional, metabolic, transcriptional, and epigenetic differences but also comprise a heterogeneous group of cells. In this review, we summarize the current findings on dynamic differentiation process during Tex heterogeneity development in cancer and chronic infection. We discuss how the responses to immunotherapy are determined by these distinct subsets and highlight prospective approaches for improving the efficacy of ICB therapy for cancer by leveraging the heterogeneity of T cells.