Comparative studies of immune-active hot and immune-deserted cold tumors are critical for identifying therapeutic targets and strategies to improve immunotherapy outcomes in cancer patients. Tumors with high tumor-infiltrating lymphocytes (TILs) are likely to respond to immunotherapy. We used the human breast cancer RNA-seq data from the cancer genome atlas (TCGA) and classified them into hot and cold tumors based on their lymphocyte infiltration scores. We compared the immune profiles of hot and cold tumors, their corresponding normal tissue adjacent to the tumor (NAT), and normal breast tissues from healthy individuals from the Genotype-Tissue Expression (GTEx) database. Cold tumors showed a significantly lower effector T cells, lower levels of antigen presentation, higher pro-tumorigenic M2 macrophages, and higher expression of extracellular matrix (ECM) stiffness-associated genes. Hot/cold dichotomy was further tested using TIL maps and H&E whole-slide pathology images from the cancer imaging archive (TCIA). Analysis of both datasets revealed that infiltrating ductal carcinoma and estrogen receptor ER-positive tumors were significantly associated with cold features. However, only TIL map analysis indicated lobular carcinomas as cold tumors and triple-negative breast cancers (TNBC) as hot tumors. Thus, RNA-seq data may be clinically relevant to tumor immune signatures when the results are supported by pathological evidence.