High level of tumor-infiltrating lymphocytes (TILs) can predict the rate of total pathological complete remission (tpCR) of breast cancer patients who receive neoadjuvant chemotherapy (NACT). This study focused on evaluating the data of patients whose primary tumor and/or lymph node metastasis show nonresponse (NR) to NACT, trying to provide a basis for the clinical decision which patients will develop NACT resistance. The study included breast cancers from 991 patients who received NACT. ROC curve analysis confirmed that TILs showed significant predictive value for NR of hormone receptor (HR)+HER2− and triple-negative breast cancer (TNBC). Among HR+HER2− breast cancer, TILs ≥ 10% was an independent predictor for low NR rate. Furthermore, positive correlation of TILs with Ki67 index and Miller-Payne grade, and negative correlation with ER and PR H-scores were only identified in this subgroup. In TNBC, TILs ≥ 17.5% was an independent predictor for low NR rate. The predictive value of low TILs on NR may facilitate to screen patients with HR+HER2− or TNBC who may not benefit from NACT. HR+HER2− breast cancer with low levels of TILs should be carefully treated with neoadjuvant chemotherapy, and other alternatives such as neoadjuvant endocrine therapy can be considered.