Immune checkpoint blockade (ICB) is currently approved for patients with triple-negative breast cancer (TNBC), whereas responses to ICB are also observed in a small subgroup of Estrogen Receptor (ER)-positive breast cancer. The cut-off for ER-positivity (≥1%) is based on likelihood of endocrine treatment response, but ER-positive breast cancer represents a very heterogeneous group. This raises the question whether selection based on ER-negativity should be revisited to select patients for ICB treatment in the context of clinical trials. Stromal tumor-infiltrating lymphocytes (sTILs) and other immune parameters are higher in TNBC compared to ER-positive breast cancer, but it is unknown whether lower ER levels are associated with more inflamed tumor microenvironments (TME). We collected a consecutive series of primary tumors from 173 HER2-negative breast cancer patients, enriched for tumors with ER expression between 1 and 99% and found levels of stromal TILs, CD8 + T cells, and PD-L1 positivity in breast tumors with ER 1–9% and ER 10–50% to be comparable to tumors with ER 0%. Expression of immune-related gene signatures in tumors with ER 1–9% and ER 10–50% was comparable to ER 0%, and higher than in tumors with ER 51–99% and ER 100%. Our results suggest that the immune landscape of ER low tumors (1–9%) and ER intermediate tumors (10–50%) mimic that of primary TNBC.