PERF: Groupby aggregations with Categorical
```
import numpy as np
np.random.seed(94356)
import pandas as pd

arr = np.arange(5).repeat(10**5)
grps = np.random.randint(0, 3, size=arr.size)

cat = pd.Categorical(arr, ordered=True)
ser = pd.Series(cat)
gb = ser.groupby(grps)

%timeit gb.min()
5.4 ms ± 212 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <- main
1.61 ms ± 14.3 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)  # <- PR

%timeit gb.first()
5.59 ms ± 136 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <- main
1.64 ms ± 14.5 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)  # <- PR
```
Thanks @jbrockmendel 