GH49560 without GH50195 (DEPR: Remove int64 uint64 float64 index part 1)
This is #49560, but without #50195, as a draft.

Locally, It looks like #49560 doesn't need #50195 to pass the tests, so I'm trying if the same is true in the github CI.

 @jbrockmendel
This didn't work, so closing.