BUG: Avoid casting Int to object in Categorical.from_codes
- [x] closes #31779
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Can you merge master? CI failure should be resolved on master thanks @dsaxton  @MeeseeksDev backport to 1.0.x