Host immune responses are a key source of selective pressure driving pathogen evolution. Emergence of many SARS-CoV-2 lineages has been associated with enhancements in their ability to evade population immunity resulting from both vaccination and infection. Here we show diverging trends of escape from vaccine-derived and infection-derived immunity for the emerging XBB/XBB.1.5 Omicron lineage. Among 31,739 patients tested in ambulatory settings in Southern California from December, 2022 to February, 2023, adjusted odds of prior receipt of 2, 3, 4, and ≥5 COVID-19 vaccine doses were 10% (95% confidence interval: 1–18%), 11% (3–19%), 13% (3–21%), and 25% (15–34%) lower, respectively, among cases infected with XBB/XBB.1.5 than among cases infected with other co-circulating lineages. Similarly, prior vaccination was associated with greater point estimates of protection against progression to hospitalization among cases with XBB/XBB.1.5 than among non-XBB/XBB.1.5 cases (70% [30–87%] and 48% [7–71%], respectively, for recipients of ≥4 doses). In contrast, cases infected with XBB/XBB.1.5 had 17% (11–24%) and 40% (19–65%) higher adjusted odds of having experienced 1 and ≥2 prior documented infections, respectively, including with pre-Omicron variants. As immunity acquired from SARS-CoV-2 infection becomes increasingly widespread, fitness costs associated with enhanced vaccine sensitivity in XBB/XBB.1.5 may be offset by increased ability to evade infection-derived host responses.