Loss of E-cadherin expression is a poor prognostic factor in patients with breast cancer. Breast cancer cells co-cultured with adipocytes reportedly promote E-cadherin attenuation and tumor progression. The current study aimed to investigate the association of reduced E-cadherin expression with adipose tissue invasion (ATI) and prognosis in breast cancer. Surgical specimens were collected from 188 women with invasive ductal carcinoma of the breast who had undergone surgery without neoadjuvant treatment. We compared E-cadherin expression in ATI and invasive front (IF) using immunohistochemistry with ImageJ. Reduced E-cadherin expression was detected not only in the ATI area but also in the IF, and the degree of reduced E-cadherin expression was positively correlated with both areas. In patients with lymph node metastasis compared to those without, E-cadherin expression was reduced and this reduction was associated with poor recurrence-free survival. We concluded that E-cadherin expression is reduced not only at the ATI area but also at the IF of the tumor. Reduced E-cadherin expression is a clear prognostic factor for breast cancer. Hence, future research is warranted for establishing an objective and quantitative E-cadherin staining assay that will allow clinical use of E-cadherin as a prognostic factor.