BUG: using loc[int] with object index
- [x] closes #19456
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This causes us to raise KeyError instead of TypeError in a couple of places, consistent with #31867.

Also note this leaves us with only one non-plotting usage of `holds_integer`, and it wouldnt surprise me if that one is subtly causing problems too.
lgtm. merge on green.