BUG: Fix Categorical.min / max bug
- [ ] closes #33450
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

pls do a PR where you test on Series as well joris pls don’t merge so fast