TST: fix flaky xfail
xfail in more specific circumstance

- [x] closes #28011
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

SGTM. https://github.com/pandas-dev/pandas/pull/27715 was backported, so we'll backport this as well.