REF: Move agg helpers into apply
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

looks good. pls rebase; 32-bit failing oddly thanks @rhshadrach 