PERF: remove categories
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

cc @phofl, per https://github.com/pandas-dev/pandas/pull/50857#issuecomment-1426898195

I dont think we should revert #50857 as that fixed a few bugs. I think this gets most of the perf back though.

No whatsnew as this was a slowdown on main only. 

```
from asv_bench.benchmarks.categoricals import RemoveCategories

b = RemoveCategories()
b.setup()

%timeit b.time_remove_categories()

# 66.4 ms ± 1.92 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  <- main
# 45.2 ms ± 1.57 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  <- PR
```
thx @lukemanley for following up so quickly looks like we got it back to the original performance, thx