DEV: remove downstream test packages from environment.yml
xref https://github.com/pandas-dev/pandas/issues/49998
There is still a doc error from using statsmodels in an old whatsnew file, will convert that to plain code block. lgtm when green > There is still a doc error from using statsmodels in an old whatsnew file, will convert that to plain code block.

I went ahead and did this, hope it's OK > I went ahead and did this, hope it's OK

Sure! Should we document in the contributing docs for now that you need to install a few additional packages to be able to run all tests in `test_downstream.py` ?  
(`pip/conda install seaborn scikit-learn statsmodels cftime pandas-datareader`)

I am not fully sure this is necessarily worth mentioning (it makes the docs more complex with yet another note, while in practice contributors won't often need this) I think it's OK to not mention this - very few contributors will be running the downstream tests anyway, and for those who do, the error message will typically make it clear what they need to install I additionally also removed `pandas-gbq`, because 1) this is actually a big dependency (with conda, indirectly it is still bringing in geopandas and its dependencies (GDAL etc)), and 2) we actually only need this for `test_downstream.py` so it _is_  a "downstream" dependency in that sense (it seems that all actual tests of `read_gbq` / `to_gbq` have been removed because this functionality is tested in pandas-gbq itself now) It would be nice to have a list of downstream dependencies somewhere. Not necessarily in the docs though. Maybe inside the test file itself?