BUG: IntervalIndex.get_indexer incorrectly matching ints to datetimes
- [x] closes #47772 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Thanks @jbrockmendel  This patch may have induced a performance regression. If it was a necessary behavior change, this may have been expected and everything is okay.

Please check the links below. If any ASVs are parameterized, the combinations of parameters that a regression has been detected for appear as subbullets.

 - [reshape.Cut.time_qcut_timedelta](https://asv-runner.github.io/asv-collection/pandas/#reshape.Cut.time_qcut_timedelta)
   - [bins=4](https://asv-runner.github.io/asv-collection/pandas/#reshape.Cut.time_qcut_timedelta?p-bins=4)
 - [reshape.Cut.time_qcut_float](https://asv-runner.github.io/asv-collection/pandas/#reshape.Cut.time_qcut_float)
   - [bins=10](https://asv-runner.github.io/asv-collection/pandas/#reshape.Cut.time_qcut_float?p-bins=10)
   - [bins=4](https://asv-runner.github.io/asv-collection/pandas/#reshape.Cut.time_qcut_float?p-bins=4)

Subsequent benchmarks may have skipped some commits. The link below lists the commits that are between the two benchmark runs where the regression was identified.

https://github.com/pandas-dev/pandas/compare/a98be065e365c3d673d759414586345b14de50ac...28f274cb107ef0a252eb4cb954c1cfbe43968395
