BUG: fix sorting with observed=True in groupby
- [x] closes #25167
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Fix bug in #25167.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25173?src=pr&el=h1) Report
> Merging [#25173](https://codecov.io/gh/pandas-dev/pandas/pull/25173?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/e3b095074c5c3b29ec25a994cd204349b872d961?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25173/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25173?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25173      +/-   ##
==========================================
+ Coverage   92.37%   92.37%   +<.01%     
==========================================
  Files         166      166              
  Lines       52408    52409       +1     
==========================================
+ Hits        48412    48413       +1     
  Misses       3996     3996
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.79% <100%> (ø)` | :arrow_up: |
| #single | `42.86% <0%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25173?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/groupby/grouper.py](https://codecov.io/gh/pandas-dev/pandas/pull/25173/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGVyLnB5) | `98.18% <100%> (+0.01%)` | :arrow_up: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25173/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `95.23% <100%> (-0.22%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25173?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25173?src=pr&el=footer). Last update [e3b0950...8ec4674](https://codecov.io/gh/pandas-dev/pandas/pull/25173?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
 # [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25173?src=pr&el=h1) Report
> Merging [#25173](https://codecov.io/gh/pandas-dev/pandas/pull/25173?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/e3b095074c5c3b29ec25a994cd204349b872d961?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25173/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25173?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25173      +/-   ##
==========================================
+ Coverage   92.37%   92.37%   +<.01%     
==========================================
  Files         166      166              
  Lines       52408    52409       +1     
==========================================
+ Hits        48412    48413       +1     
  Misses       3996     3996
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.79% <100%> (ø)` | :arrow_up: |
| #single | `42.86% <0%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25173?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/groupby/grouper.py](https://codecov.io/gh/pandas-dev/pandas/pull/25173/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGVyLnB5) | `98.18% <100%> (+0.01%)` | :arrow_up: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25173/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `95.23% <100%> (-0.22%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25173?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25173?src=pr&el=footer). Last update [e3b0950...8ec4674](https://codecov.io/gh/pandas-dev/pandas/pull/25173?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
 Closing as stale ping if you'd like to continue