BUG/API: preserve non-nano in factorize/unique
- [ ] closes #51961 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The ugly bit here is that it causes `to_datetime` to behave slightly differently with cache=True/False in some cases.  I know @mroeschke worked to avoid that in #17077, so it may be a deal-breaker.

But it fixes #51961 (though doesn't yet have a targeted test) so may be worth prioritizing (cc @MarcoGorelli)

(Update) also fixes Series[non-nano].factorize which currently raises AssertionError
> But it fixes https://github.com/pandas-dev/pandas/issues/51961 (though doesn't yet have a targeted test) so may be worth prioritizing

Fine with me to have this fix despite the cache=False bug. Could you add a test for the above too? Thanks @jbrockmendel  Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 6a13450cec5996cc14f722069ef9be150f034e03
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #51978: BUG/API: preserve non-nano in factorize/unique'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-51978-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #51978 on branch 2.0.x (BUG/API: preserve non-nano in factorize/unique)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                 Mind creating the backport PR @jbrockmendel ?