CI: Fix pytorch installation error
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

The conda installation expects mkl locally, which does not work . So installing via pip for now to get ci back to green
thanks @phofl 