BUG: fixes #53935 Categorical order lost after call to remove_categories 
- [x] closes https://github.com/pandas-dev/pandas/issues/53935
- [x]  [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
 - [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
 - [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
 - [x] Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

I simply put an if/else in the `remove_categories` method.  If `ordered=True` it will pass `sort=False` into the `difference1 method.


Thanks @paulreece  Great working with you, @mroeschke!