BUG: Fix Categorical use_inf_as_na bug
- [x] closes #33594
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

thanks @dsaxton 

if you can push the followon when you can to consolidate code here would be great xref #33300 to track.
