Remove xlrd benchmark after xlrd was updated to 2.0.1 on conda-forge
- [x] closes #39076
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Is pinning here the right thing to do in environment.yml?

cc @jreback 
Alternatively we could remove this benchmark Done Have to backport this too. great, backporting as well to avoid 1.2.x CI failures @jreback greenish based on the changes here. Azure will take a while because of the new master commits thanks @phofl  @meeseeksdev backport 1.2.x