BUG/PERF: Series(category).replace
- [x] closes #50872
- [x] closes #46884
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.0.0.rst` file if fixing a bug or adding a new feature.

Redo of #49404 with fix that was reverted in #50848
Thanks for resubmitting @lukemanley  Could this have caused a performance regression? https://asv-runner.github.io/asv-collection/pandas/#categoricals.RemoveCategories.time_remove_categories