BackgroundBreast cancer presents as one of the top health threats to women around the world. Myeloid cells are the most abundant cells and the major immune coordinator in breast cancer tumor microenvironment (TME), target therapies that harness the anti-tumor potential of myeloid cells are currently being evaluated in clinical trials. However, the landscape and dynamic transition of myeloid cells in breast cancer TME are still largely unknown.MethodsMyeloid cells were characterized in the single-cell data and extracted with a deconvolution algorithm to be assessed in bulk-sequencing data. We used the Shannon index to describe the diversity of infiltrating myeloid cells. A 5-gene surrogate scoring system was then constructed and evaluated to infer the myeloid cell diversity in a clinically feasible manner.ResultsWe dissected the breast cancer infiltrating myeloid cells into 15 subgroups including macrophages, dendritic cells (DCs), and monocytes. Mac_CCL4 had the highest angiogenic activity, Mac_APOE and Mac_CXCL10 were highly active in cytokine secretion, and the DCs had upregulated antigen presentation pathways. The infiltrating myeloid diversity was calculated in the deconvoluted bulk-sequencing data, and we found that higher myeloid diversity was robustly associated with more favorable clinical outcomes, higher neoadjuvant therapy responses, and a higher rate of somatic mutations. We then used machine learning methods to perform feature selection and reduction, which generated a clinical-friendly scoring system consisting of 5 genes (C3, CD27, GFPT2, GMFG, and HLA-DPB1) that could be used to predict clinical outcomes in breast cancer patients.ConclusionsOur study explored the heterogeneity and plasticity of breast cancer infiltrating myeloid cells. By using a novel combination of bioinformatic approaches, we proposed the myeloid diversity index as a new prognostic metric and constructed a clinically practical scoring system to guide future patient evaluation and risk stratification.Graphical AbstractSupplementary InformationThe online version contains supplementary material available at 10.1186/s13058-023-01669-6.