REF: smarter NaN handling in remove_unused_levels()
- [x] tests passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Sorry for the bad timing @jreback , only after you merged #18426 this simpler way to proceed came to my mind.

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=h1) Report
> Merging [#18438](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/fedc503f0d25b04850a931398fc0556d892d69ea?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18438/graphs/tree.svg?height=150&width=650&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18438      +/-   ##
==========================================
- Coverage   91.35%   91.33%   -0.02%     
==========================================
  Files         163      163              
  Lines       49691    49688       -3     
==========================================
- Hits        45397    45385      -12     
- Misses       4294     4303       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.14% <100%> (-0.01%)` | :arrow_down: |
| #single | `39.67% <0%> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `96.4% <100%> (-0.01%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.8% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=footer). Last update [fedc503...d08fd29](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
 # [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=h1) Report
> Merging [#18438](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/fedc503f0d25b04850a931398fc0556d892d69ea?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18438/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18438      +/-   ##
==========================================
- Coverage   91.35%   91.33%   -0.02%     
==========================================
  Files         163      163              
  Lines       49691    49688       -3     
==========================================
- Hits        45397    45385      -12     
- Misses       4294     4303       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.14% <100%> (-0.01%)` | :arrow_down: |
| #single | `39.67% <0%> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `96.4% <100%> (-0.01%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.8% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=footer). Last update [fedc503...d08fd29](https://codecov.io/gh/pandas-dev/pandas/pull/18438?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
 thanks! @toobaz 

separately, it might be nice to see (on a bigger MI), how much time this actually takes, if its small then it might be theoretically feasible to actually do this (or at least check whether we need to do this). > might be theoretically feasible to actually do this

Do you mean to implicitly do this (e.g. when slicing)? yep - seeing if this is feasible But do we actually want this? E.g. a categorical does not loose categories when it is sliced. it’s actuaiky confusing to users
if it were cheap i think we would (but i don’t think it’s cheap)