A variety of variables, such as microsatellite instability or inflammatory mediators, are critical players in the development and progression of colorectal cancer (CRC). Natural killer (NK) and natural killer T (NKT) cells are involved in the prognoses of CRC. Immunological components of the tumor microenvironment (TME) impact cancer progression and therapeutic responses. We report that CRC patients with higher frequencies of tumor-infiltrating PD-1+NK and NKT cells had significantly longer disease-free survival (DFS) than patients with lower frequencies. In agreement with that, patients with higher frequencies of tumor-infiltrating PD-1−NK and NKT cells showed shorter DFS. There were no significant associations between tumor-infiltrating PD-1+TIM-3+, PD-1+TIGIT+, PD-1+ICOS+, PD-1+LAG-3+NK cells, and PD-1+TIM-3+, PD-1+TIGIT+, and PD-1+LAG-3+NKT cells with DFS. This study highlights the significance of PD-1 expression on tumor-infiltrating NK and NKT cells and its association with disease prognoses in CRC patients.Supplementary InformationThe online version contains supplementary material available at 10.1007/s00262-022-03337-8.