The relationship between total lymphocyte counts (TLCs) and survival is not well documented in rectal cancer. This study aimed to investigate the association between TLCs and disease-free survival (DFS) and identify factors associated with lymphopenia in locally advanced rectal cancer patients receiving chemoradiotherapy. Thirty-six patients with locally advanced rectal cancer were retrospectively analyzed. TLCs were evaluated before surgery (pre-S), before radiotherapy (pre-RT), and during concurrent chemoradiotherapy (CCRT). The relationship between TLCs and DFS was analyzed by univariate and multivariate analysis. Potential clinical factors associated with lymphopenia were also evaluated. Median TLC declined significantly during radiotherapy. Severe lymphopenia during CCRT was significantly associated with poorer DFS on Kaplan–Meier analysis (p = 0.01), univariate regression analysis (p = 0.036), and multivariate regression analysis (p = 0.038). Pre-S TLCs (p = 0.009) and pre-RT TLCs (p = 0.042) were significantly associated with severe lymphopenia on univariate regression analysis; however, only pre-S TLCs (p = 0.026) were significantly associated with severe lymphopenia on multivariate regression analysis. Severe lymphopenia was a predictor of poorer DFS in patients with locally advanced rectal cancer receiving adjuvant chemoradiotherapy. Pre-S TLCs were predictors of severe lymphopenia. Further study is warranted to reduce the rate of severe lymphopenia.