REF: unify casting logic in Categorical.__init__
Big picture, trying to call maybe_infer_to_datetimelike in fewer places so we can simplify scattered datetimelike casting.

pls have a check on perf, IIRC we were pretty sensitive here (prob unchanged but who knows)