Improve to_datetime bounds checking
to_datetime(errors='raise') is supposed to raise an exception on out-of-bounds input.  However, it uses rounding to integer for this bounds checking but not the actual conversion, and hence, for input just outside the bounds it may instead return NaT ([on x86](https://buildd.debian.org/status/fetch.php?pkg=pandas&arch=all&ver=1.5.2%2Bdfsg-1&stamp=1669856039&raw=0)) or output clipped to the bounds ([on arm](https://buildd.debian.org/status/fetch.php?pkg=pandas&arch=arm64&ver=1.5.2%2Bdfsg-1&stamp=1669872650&raw=0)).

It also assumes that converting NaN to int gives NaT, which [may not be true on unusual hardware](https://buildd.debian.org/status/fetch.php?pkg=pandas&arch=riscv64&ver=1.4.3%2Bdfsg-1&stamp=1659694881&raw=0).  (This was how I originally noticed the problem, as Debian tries to build packages on a wide range of hardware.)

This patch fixes both issues.  (In the Debian logs linked here, search for 'near-limits test', the first run is unpatched 1.3.5, the second run is patched 1.5.x.)

Its effect on speed is unclear: it is [sometimes faster](https://buildd.debian.org/status/fetch.php?pkg=pandas&arch=amd64&ver=1.5.1%2Bdfsg-1&stamp=1667870121&raw=0) and [sometimes slower](https://buildd.debian.org/status/fetch.php?pkg=pandas&arch=amd64&ver=1.5.2%2Bdfsg-1&stamp=1669859491&raw=0), possibly because other load on the build machines varies.

- [this bug does not appear to be previously reported ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [N/A ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this. Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen. I'll add a commit here when I get a chance (hopefully early next week), I think we should ship this, and @rebecca-palmer did some seriously good work here Fixed conflicts (which was quite a large change, so wait for the CI before merging - as well as #48705 having done part of this, #50263 moved the code being patched to a non-Cython context, and #50301 forbids the case I was using as a test).

Unrelated to my changes, but it looks like #50263 may have accidentally undone the effect of #50301 for some input types - @jbrockmendel ? > Unrelated to my changes, but it looks like https://github.com/pandas-dev/pandas/pull/50263 may have accidentally undone the effect of https://github.com/pandas-dev/pandas/pull/50301 for some input types - @jbrockmendel ?

That's entirely plausible.  Would be happy to see a PR fixing that. That CI fail looks like the same one main then had - retrying. Fixed, but also removed the unnecessary copy, so wait for the CI to check that it really is unnecessary. Does the Windows CI timing out suggest there's a hang bug in this, or does that just happen sometimes?

(Numpy Dev also fails on main, for reasons that are nothing to do with pandas: numpy/numpy#23033) Thanks @rebecca-palmer 