CI: remove test cases which would have globbed pandas.tests
This came up here https://github.com/pandas-dev/pandas/pull/54365#issuecomment-1664639342

I've removed the test cases which would've ended up triggering globbing of pandas/tests, this should remove the noise observed in that PR
Thanks for the quick fix @MarcoGorelli 