Immune checkpoint blockade (ICB) treatment of hepatocellular carcinoma (HCC) patients with hepatitis B virus (HBV) infection may activate viral-specific T cells to attack HBV infected hepatocytes and thus induce immune-related liver injury. Therefore, it is important to deeply understand the impacts of HBV infection on HCC immune microenvironment in order to better design effective immunotherapies for HBV+(HBV infected) HCC patients. Here, We performed cytometry by time-of-flight (CyTOF) analyses to characterize the distinct immune compositions of HCC tumors, tumor borders, and their associations with HCC/HBV related clinical characteristics. We identified 31 distinct immune clusters and found significant associations between immune signatures with clinicopathological features of HCC. We further revealed the HBV infection had more effects on shaping immune compositions in tumor borders than in tumors, with the significant enrichment of HBV-specific PD-1+CD8+tissue-resident memory T (TRM) cells in tumor borders of HBV+patients. We confirmed this subset with a more exhausted phenotype and respond more actively under anti-PD-L1 treatment, suggesting its involvement in immune-related liver injury induced by ICB treatment to HBV+HCC patients. Our study shows it may be necessary to consider antiviral prophylaxis for HBV+HCC patients receiving ICB treatment.Supplementary InformationThe online version contains supplementary material available at 10.1186/s13046-023-02710-4.