BUG: to_datetime with floats and unit not matching Timestamp
- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The edited sas test is unfortunate but i think OK overall.  Will revisit once #56014 is addressed.  Also i expect that behavior to change in the not-too-distant future to return non-nano anyway.

In a follow-up I'll try to de-duplicate cast_from_unit and cast_from_unit_vectorized.
Thanks @jbrockmendel  This seems to have caused a pretty bad performance regression

https://asv-runner.github.io/asv-collection/pandas/#inference.ToDatetimeFromIntsFloats.time_nanosec_float64