CI: More targeted pyarrow version testing
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature

Hoping to supersede https://github.com/pandas-dev/pandas/pull/46386. cc @lithomas1 

While the job queue bandwidth is larger now, probably don't need to test different pyarrow versions with different Python versions. Instead:

* PY38 will test pyarrow 5, 6, 7
* PY39 and PY310 will just test pyarrow 5
> Thanks for picking this up! Do you know if we should remove pyarrow from the env files, given that we're installing it manually here?

I don't think we can since the Azure Windows/MacOS builds use these files and don't install pyarrow manually @meeseeksdev backport 1.4.x