Immunotherapy has revolutionized cancer care in the past decade. Treatment with immune checkpoint inhibitors has demonstrated promising clinical activity against tumors. However, only a subset of patients responds to these treatments, limiting their potential benefit. Efforts to understand, predict, and overcome the lack of response in patients, have thus far focused mainly on the tumor immunogenicity and the quantity and characteristics of tumor-infiltrating T cells, since these cells are the main effectors of immunotherapies. However, recent comprehensive analyses of the tumor microenvironment (TME) in the context of immune checkpoint blockade (ICB) therapy have revealed critical functions of other immune cells in the effective anti-tumor response, highlighting the need to account for complex cell-cell interaction and communication underlying clinical outputs. In this perspective, I discuss the current understanding of the crucial roles of tumor-associated macrophages (TAMs) in the success of T cell-directed immune checkpoint blockade therapies, as well as the present, and the future of clinical trials on combinatorial therapies targeting both cell types.