PERF: Improve performance for arrow engine and dtype_backend=pyarrow for datetime conversion
- [x] closes #52546 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

should we exclude any other arrow types from the conversion?
I'm not sure we should backport this to 2.0.x. 

Parse dates support is still one of the more broken things for the pyarrow parser, so a lot of the date handling stuff could be subject to change.

(I am planning to unstale #50056 once my other pyarrow PR is merged). We almost certainly should backport this. dtype_backend is new, so I am not really concerned with compatibility issues right now (this is only hit if dtype_backend is set).

Reasons to backport:

- if parse_dates is set and arrow already guessed the format we are still converting back to NumPy (shouldn't do this)
- Performance! I have an example from medium where main runs 7 seconds and this version runs 0.025 seconds. We should avoid calling to_datetime for ArrowExtensionArrays as far as possible and since this makes stuff actually more consistent (you get an arrow dtype instead of a NumPy dtype) I don't see a reason why we shouldn't backport. > Reasons to backport:
> 
> * if parse_dates is set and arrow already guessed the format we are still converting back to NumPy (shouldn't do this)
> * Performance! I have an example from medium where main runs 7 seconds and this version runs 0.025 seconds. We should avoid calling to_datetime for ArrowExtensionArrays as far as possible and since this makes stuff actually more consistent (you get an arrow dtype instead of a NumPy dtype) I don't see a reason why we shouldn't backport.

Thanks for clarifying. This is fine then, since it fixes the issue with us ending up with numpy dtype (when we should get the Arrow dtype).

One last thing, can you add a whatsnew note since the performance difference is pretty big (as you noted)? I'll phrase it more like a bug fix, but will mention performance as well