CI: Install more packages from conda-forge
- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Let's see how this goes. Not sure if any of those pip installs are there on purpose or just because at the time of writing no conda-forge package was available.
Test failures seem to be unrelated? Oops I undid my Zstandard removal with the rebase. Will rebase and fix  thanks @jonashaag 

