PERF: IntegerIndex._shallow_copy (and by extension, indexing)
```
In [2]: idx = pd.Index(range(1000)).insert(-1, 1)                                                                                                                                                                                                 
In [3]: %timeit idx[:5]                                                                                                                                                                                                                           
2.84 µs ± 97.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- PR
24.5 µs ± 292 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- master

In [4]: idx2 = idx.astype('uint64')                                                                                                                                                                                                               
In [5]: %timeit idx2[5:]                                                                                                                                                                                                                          
2.99 µs ± 331 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- PR
25.8 µs ± 957 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- master
```

Motivated by profiling in #32086 and seeing that _shallow_copy_with_infer was taking over a third of the runtime in the relevant benchmark.
likely we don't have these benchmarks directly, maybe add if you can.