REF: to_datetime handle non-object cases outside cython
- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Thanks @jbrockmendel  @jbrockmendel this looks like a likely culprit for https://asv-runner.github.io/asv-collection/pandas/#inference.ToDatetimeFromIntsFloats.time_sec_uint64

The regression occurred between d3e5e05 and 5f3c29e, so not totally sure that this is the problem, but the asv hits the code paths introduced here More or less all the time is spend in astype_overflowsafe Yah I noticed that too.  Reverting wouldn't be the worst thing, thought might be worth checking that the old version didn't allow silent overflows