CI: xfail intermittently-failing tests
- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Usually I avoid `strict=False` like the plague, but these are a PITA recently.
lgtm. merge on green (also if you can make an issue to revert this)