BUG: loc-indexing with a CategoricalIndex with non-string categories
- [x] closes #17569
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Bug when indexing with ``.loc`` and the index is a CategoricalIndex with integer or float categories.
Hello @topper-123! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-11 07:13:21 UTC Ill be abroad untill the weekend, will look at it then. I think it would cause a error with categories made from datetimeindex, but I’ll check. I`ve updated the PR: I´ve removed _defer_to_indexing and added some tests. I've updated and use the dtypes from pandas/conftest.py instead. thanks @topper-123 very nice!