REF: simplify _cython_agg_blocks
Orthogonal to #35839, though a rebase will be needed.

cc @TomAugspurger did you already try this in #31616?  If so, we need to identify a test case in which this doesnt work
No, I don't think I tried anything like this. thanks, yet another reason to do the these ops in internals ....