REF: de-duplicate object-dtype handling
We'll also use this for the categorical case following #32167.
you want to merge #32167 first? > you want to merge #32167 first?

either order is going to require a rebase i think