Tumor-infiltrating immune cells and fibroblasts are significant components of the tumor microenvironment (TME) of pancreatic ductal adenocarcinoma (PDAC), and they participate in tumor progression as closely as tumor cells. However, the relationship between the features of the TME and patient outcomes and the interactions among TME components are still unclear. In this study, we evaluated the PDAC TME in terms of the quantity and location of cluster of differentiation (CD)4+T cells, CD8+T cells, macrophages, stromal maturity, and tumor-stroma ratio (TSR), as evaluated by immunohistochemical staining of serial whole-tissue sections from 116 patients with PDAC. The density of T cells and macrophages (mainly activated macrophages) was significantly higher at the invasive margins (IMs) than at the tumor center (TC). CD4+T cells were significantly association with all the other tumor-associated immune cells (TAIs) including CD8, CD68 and CD206 positive cells. Tumors of the non-mature (intermediate and immature) stroma type harbored significantly more CD8+T cells at the IMs and more CD68+macrophages at the IMs and the TC. The density of CD4+, CD8+, and CD206+cells at the TC; CD206+cells at the IMs; and tumor-node-metastasis (TNM) staging were independent risk factors for patient outcomes, and the c-index of the risk nomogram for predicting the survival probability based on the TME features and TNM staging was 0.772 (95% confidence interval: 0.713–0.832). PDAC harbored a significantly immunosuppressive TME, of which the IMs were the hot zones for TAIs, while cells at the TC were more predictive of prognosis. Our results indicated that the model based on the features of the TME and TNM staging could predict patient outcomes.Supplementary InformationThe online version contains supplementary material available at 10.1007/s12672-023-00744-w.