It is well-established that tumor antigens and molecules expressed and secreted by cancer cells trigger innate and adaptive immune responses. These two types of anti-tumor immunity lead to the infiltration of the tumor’s microenvironment by immune cells with either regulatory or cytotoxic properties. Whether this response is associated with tumor eradication after radiotherapy and chemotherapy or regrowth has been a matter of extensive research through the years, mainly focusing on tumor-infiltrating lymphocytes and monocytes and their subtypes, and the expression of immune checkpoint and other immune-related molecules by both immune and cancer cells in the tumor microenvironment. A literature search has been conducted on studies dealing with the immune response in patients with rectal cancer treated with neoadjuvant radiotherapy or chemoradiotherapy, assessing its impact on locoregional control and survival and underlying the potential role of immunotherapy in the treatment of this cancer subtype. Here, we provide an overview of the interactions between local/systemic anti-tumor immunity, cancer-related immune checkpoint, and other immunological pathways and radiotherapy, and how these affect the prognosis of rectal cancer patients. Chemoradiotherapy induces critical immunological changes in the tumor microenvironment and cancer cells that can be exploited for therapeutic interventions in rectal cancer.