The COVID-19 response strategies in Chinese mainland were recently adjusted due to the reduced pathogenicity and enhanced infectivity of Omicron subvariants. In Chengdu, China, an infection wave was predominantly induced by the BA.5 subvariant. It is crucial to determine whether the hybrid anti-SARS-CoV-2 immunity following BA.5 infection, coupled with a variety of immune background, is sufficient to shape the immune responses against newly emerged Omicron subvariants, especially for XBB lineages. To investigate this, we collected serum and nasal swab samples from 108 participants who had been infected in this BA.5 infection wave, and evaluated the neutralization against pseudoviruses. Our results showed that convalescent sera from individuals, regardless of vaccination history, had remarkably compromised neutralization capacities against the newly emerged XBB and XBB.1.5 subvariants. Although post-vaccination with BA.5 breakthrough infection slightly elevated plasma neutralizing antibodies against a part of pseudoviruses, the neutralization activities were remarkably impaired by XBB lineages. Furthermore, we analyzed the impacts of the number of vaccinations, age, and sex on the humoral and cellular immune response after BA.5 infection. Our findings suggest that the neutralization against XBB lineages that elicited by current hybrid immunity after BA.5 infection, are remained at low levels, indicating an urgent need for the development of next-generation of COVID-19 vaccines that designed based on the XBB sub-lineages and other future variants.