PERF: Index.difference
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.


asv continuous -f 1.1 upstream/main perf-index-difference -b index_object.SetDisjoint

```
       before           after         ratio
-      7.97±0.4ms       3.16±0.4ms     0.40  index_object.SetDisjoint.time_datetime_difference_disjoint
```

asv continuous -f 1.1 upstream/main perf-index-difference -b multiindex_object.Difference

```
       before           after         ratio
-      3.55±0.7ms       1.29±0.3ms     0.36  multiindex_object.Difference.time_difference('int')
-      3.79±0.5ms       1.20±0.2ms     0.32  multiindex_object.Difference.time_difference('datetime')
```
Thanks @lukemanley  This patch may have induced a performance regression. If it was a necessary behavior change, this may have been expected and everything is okay.

Please check the links below. If any ASVs are parameterized, the combinations of parameters that a regression has been detected for appear as subbullets.

 - [index_object.SetDisjoint.time_datetime_difference_disjoint](https://asv-runner.github.io/asv-collection/pandas/#index_object.SetDisjoint.time_datetime_difference_disjoint)
 - [multiindex_object.Difference.time_difference](https://asv-runner.github.io/asv-collection/pandas/#multiindex_object.Difference.time_difference)
   - [dtype='int'](https://asv-runner.github.io/asv-collection/pandas/#multiindex_object.Difference.time_difference?p-dtype=%27int%27)
   - [dtype='datetime'](https://asv-runner.github.io/asv-collection/pandas/#multiindex_object.Difference.time_difference?p-dtype=%27datetime%27)
   - [dtype='ea_int'](https://asv-runner.github.io/asv-collection/pandas/#multiindex_object.Difference.time_difference?p-dtype=%27ea_int%27)
   - [dtype='string'](https://asv-runner.github.io/asv-collection/pandas/#multiindex_object.Difference.time_difference?p-dtype=%27string%27)

Subsequent benchmarks may have skipped some commits. The link below lists the commits that are between the two benchmark runs where the regression was identified.

https://github.com/pandas-dev/pandas/compare/79067a76adc448d17210f2cf4a858b0eb853be4c...0bdbc44babac09225bdde02b642252ce054723e3
