PERF/REGR: symmetric_difference revert most of #41833
- [ ] closes #42239
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

aimed at #42239
```
(pandas-dev) simon@T3630:~/pandas/asv_bench (perf-symmetric_difference)$ asv compare v1.3.0.dev0 4c878d244a369cc7689f6c01c1a8c4d77b405cc5 |grep SetOperations.time_operation.*symmetric_difference
      15.0±0.09ms         16.1±2ms     1.08  index_object.SetOperations.time_operation('date_string', 'symmetric_difference')
+     3.75±0.03ms      4.24±0.07ms     1.13  index_object.SetOperations.time_operation('datetime', 'symmetric_difference')
      2.84±0.04ms       2.91±0.3ms     1.03  index_object.SetOperations.time_operation('int', 'symmetric_difference')
       13.8±0.2ms       14.9±0.2ms     1.07  index_object.SetOperations.time_operation('strings', 'symmetric_difference')
+         422±3ms          670±6ms     1.59  multiindex_object.SetOperations.time_operation('monotonic', 'datetime', 'symmetric_difference')
+      42.2±0.2ms       60.1±0.5ms     1.42  multiindex_object.SetOperations.time_operation('monotonic', 'int', 'symmetric_difference')
+        42.2±2ms       48.6±0.6ms     1.15  multiindex_object.SetOperations.time_operation('monotonic', 'string', 'symmetric_difference')
+         423±3ms          652±7ms     1.54  multiindex_object.SetOperations.time_operation('non_monotonic', 'datetime', 'symmetric_difference')
+      42.6±0.5ms       58.9±0.4ms     1.38  multiindex_object.SetOperations.time_operation('non_monotonic', 'int', 'symmetric_difference')
+        42.4±1ms       48.3±0.5ms     1.14  multiindex_object.SetOperations.time_operation('non_monotonic', 'string', 'symmetric_difference')
``` comparing this PR to master

```
(pandas-dev) simon@T3630:~/pandas/asv_bench (perf-symmetric_difference)$ asv compare master 4c878d244a369cc7689f6c01c1a8c4d77b405cc5 |grep SetOperations.time_operation.*symmetric_difference
-      31.1±0.7ms         16.1±2ms     0.52  index_object.SetOperations.time_operation('date_string', 'symmetric_difference')
-      7.25±0.1ms      4.24±0.07ms     0.58  index_object.SetOperations.time_operation('datetime', 'symmetric_difference')
-      5.16±0.3ms       2.91±0.3ms     0.57  index_object.SetOperations.time_operation('int', 'symmetric_difference')
-      28.7±0.7ms       14.9±0.2ms     0.52  index_object.SetOperations.time_operation('strings', 'symmetric_difference')
-      1.13±0.01s          670±6ms     0.59  multiindex_object.SetOperations.time_operation('monotonic', 'datetime', 'symmetric_difference')
-         105±3ms       60.1±0.5ms     0.57  multiindex_object.SetOperations.time_operation('monotonic', 'int', 'symmetric_difference')
-      81.5±0.5ms       48.6±0.6ms     0.60  multiindex_object.SetOperations.time_operation('monotonic', 'string', 'symmetric_difference')
-      1.14±0.03s          652±7ms     0.57  multiindex_object.SetOperations.time_operation('non_monotonic', 'datetime', 'symmetric_difference')
-         103±1ms       58.9±0.4ms     0.57  multiindex_object.SetOperations.time_operation('non_monotonic', 'int', 'symmetric_difference')
-        81.9±1ms       48.3±0.5ms     0.59  multiindex_object.SetOperations.time_operation('non_monotonic', 'string', 'symmetric_difference')

```

updating to close issue odd. https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=62375&view=results

will merge as it appears tests were successful Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 be2efa98dd97f86809d741070757fd1cbe425fc2
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #42318: PERF/REGR: symmetric_difference revert most of #41833'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-42318-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #42318 on branch 1.3.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                 > updating to close issue

on 1.3.x Index.symmetric_difference is overridden for MultiIndex so we will need to recheck timings