REGR: Index.map adding back tz to tz-agnostic result
- [x] closes #57192 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

It seems to me we should skip casting back to the caller's dtype when the caller dtype is datetime, since these are reliably represented as objects (unlike ints, which loses the size). But datetimes/tzs are definitely not in my wheelhouse.

cc @jbrockmendel 
I'm pretty sure the problem is in DatetimeArray._from_scalars, where we need to check that dont have tzawareness-mismatch.  instead of 

```
if lib.infer_dtype(scalars, skipna=True) not in ["datetime", "datetime64"]:
    raise ValueError
```

something like

```
if isinstance(dtype, DatetimeTZDtype) and not lib.is_datetime_with_singletz_array(scalars);
     raise ValueError
elif isinstance(dtype, np.dtype) and not lib.is_naive_datetime_array(scalars):
    raise ValueError
```

is_naive_datetime_array doesn't actually exist but seems like it should. Thanks @jbrockmendel - this is much better. One thing I'm running into is that `is_datetime_with_singletz_array` doesn't check if any of the values are actually datetimes (will always return True when there are no datetimes), so I think this still needs to have the line with `lib.infer_dtype(scalars, skipna=True)` or maybe use `lib.is_datetime_or_datetime64_array`? Any recommendations? I'm trying to get rid of uses of lib.infer_dtype as it is very rarely the right tool to reach for.

Seems like `is_datetime_with_singletz_array(values) and not isna(values).all()` is the simplest way to get what you've described.  Could try to combine those into a single function i guess but i wouldn't bother at this point.