PERF: Fix performance regression in get_loc of IntervalIndex
- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


caused in https://github.com/pandas-dev/pandas/commit/184e1674

No need to call validate here

https://asv-runner.github.io/asv-collection/pandas/#indexing.IntervalIndexing.time_loc_list
pandas/core/arrays/interval.py:729: error: Argument 2 to "_simple_new" of "IntervalArray" has incompatible type "Union[Period, Timestamp, Timedelta, NaTType, DatetimeArray, TimedeltaArray, ndarray[Any, Any]]"; expected "Union[Union[DatetimeArray, TimedeltaArray], ndarray[Any, Any]]"  [arg-type]
