CLN: clean up groupby / categorical
xref #21151

cc @WillAyd 
cc @jschendel  # [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21753?src=pr&el=h1) Report
> Merging [#21753](https://codecov.io/gh/pandas-dev/pandas/pull/21753?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d24a9507ba539f455d9b90885edf098c7dc93e99?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `96.29%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21753/graphs/tree.svg?token=eZ4WkYLtcO&height=150&width=650&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/21753?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #21753      +/-   ##
==========================================
- Coverage   91.93%   91.93%   -0.01%     
==========================================
  Files         159      160       +1     
  Lines       49733    49737       +4     
==========================================
+ Hits        45721    45724       +3     
- Misses       4012     4013       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.3% <96.29%> (-0.01%)` | :arrow_down: |
| #single | `41.99% <0%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21753?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/21753/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.93% <ø> (-0.09%)` | :arrow_down: |
| [pandas/core/groupby/groupby.py](https://codecov.io/gh/pandas-dev/pandas/pull/21753/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGJ5LnB5) | `92.65% <100%> (-0.02%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/21753/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `95.65% <95.65%> (ø)` | |
| [pandas/core/indexes/category.py](https://codecov.io/gh/pandas-dev/pandas/pull/21753/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9jYXRlZ29yeS5weQ==) | `97.01% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21753?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21753?src=pr&el=footer). Last update [d24a950...5783ce1](https://codecov.io/gh/pandas-dev/pandas/pull/21753?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
 cc @TomAugspurger ok?