REV: Revert #40510
@jorisvandenbossche reports that #40510 was performance-hurting, so this rolls it back until we can sort out why that is.
Thanks @jbrockmendel !

I checked some of the benchmarks that were affected, and they improved again, see eg https://pandas.pydata.org/speed/pandas/#groupby.GroupByCythonAgg.time_frame_agg?python=3.8&Cython=0.29.21&jinja2=&p-method='mean' 
(this one might also be a good case to check what was actually the cause, if you're interested)