REF: Refactor CI env file creation and reduce pyarrow testing
- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Curious, what is the benefit for having a script generate the dependency files at this point? Personally I slightly prefer the explicitness of just having the yamls.

I could see the benefit of a script/job that would generate conda lock files > Curious, what is the benefit for having a script generate the dependency files at this point? Personally I slightly prefer the explicitness of just having the yamls.
> 
> I could see the benefit of a script/job that would generate conda lock files

It's hard to keep everything in sync with the action-3x yamls. 

They are all essentially the same, except for the Python version, and when a new one is added, it is easy for dependencies to disappear if they didn't work (you have to be careful that the dependencies match when reviewing the PR, which is burdensome). conda lock seems cool, but I don't think it would be able to handle generate an environment file for different Python versions.

Might be worth it anyways, since pinning dependencies would make CI a lot more reliable. Agreed the file are mostly the same besides python versions, but IMO the maintenance of them hasn't been too burdensome since the dependencies don't change often. We would be trading maintenance of these files with maintenance of this script.

> conda lock seems cool, but I don't think it would be able to handle generate an environment file for different Python versions.

I was working on a POC of generating conda-lock files for a platform + python version on my repo: https://github.com/mroeschke/pandas/pull/13/files. conda-lock would also be nice for the benefit of eliminating environment solving time too The pain is mostly when supporting the newest Python, say Python 3.11.

A lot of packages like numba are slower to adopt the newest Python, and sometimes people forget to comment those dependencies out instead of deleting them. This is hard to spot when reviewing.

In general, I'd say that the script is pretty low maintenance, since the conda yaml format doesn't change.

It's also annoying, when it comes time to bump things like Cython versions (since CI won't fail, since we specify a lower bound, and CI will still pull the latest version even though the lower bound isn't correct).

I guess the main burden would be if you wanted to parametrize over the version of other packages in addition to pyarrow. Fine with closing, didn't realize people were creating envs from this.