Backport PR #42261 (BUG: Fix to_datetime() cache behaviour to not omit duplicated output values)

Backport PR #42261
@phofl can ignore the `test_rename_inplace` failure.

Does this fix seem reasonable and OK to backport. Thanks @phofl 