CI: Don't test min pyarrow version with multiple python versions
cc @phofl 

After the minimum pyarrow version was bumped to 7, we shouldn't need to test this over multiple python versions anymore.

Additionally, uses PY3.11 to test multiple pyarrow versions instead of PY3.10
Looks like the non flaky failures are due to #51006 so merging to reduce the number of builds running