BUG: Regression in logical ops raising ValueError with Categorical columns with unused categories
- [x] closes #38367
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

@simonjayhawkins 

This would fix the issue, but is pretty dirty. If we decide to merge this, we should remove this again with #38140 Also test would have to be adjusted (expected remove unused category)
thanks @phofl  @meeseeksdev backport 1.2.x