Breast milk represents the optimal source of feeding for newborns, in terms of nutritional compounds and as it provides immunological, metabolic, organic, and neurological well-being. As a complex biological fluid, it consists not only of nutritional compounds but also contains environmental contaminants. Formulas through production, contact with bottles and cups, and complementary feeding can also be contaminated. The current review focuses on endocrine-disrupting chemicals, and made-man xenoestrogens present in the environment and both commonly present in food sources, agricultural practices, packaging, consumer products, industry, and medical care. These contaminants are transferred by passive diffusion to breast milk and are delivered during breastfeeding. They mainly act by activating or antagonizing hormonal receptors. We summarize the effects on the immune system, gut microbiota, and metabolism. Exposure to endocrine-disrupting chemicals and indirect food additives may induce tissue inflammation and polarize lymphocytes, increase proinflammatory cytokines, promote allergic sensitization, and microbial dysbiosis, activate nuclear receptors and increase the incidence of allergic, autoimmune, and metabolic diseases. Breast milk is the most important optimal source in early life. This mini-review summarizes current knowledge on environmental contaminants and paves the way for strategies to prevent milk contamination and limit maternal and infant exposure during pregnancy and the first months of life.