Alternate Ubuntu CI fix
@MarcoGorelli @lithomas1 @jbrockmendel 

I think if we clean up some of this dead code we can support a clean build on newer gcc. Lots of layers of indirection to sift through so not entirely clear on what's happening, but I think gcc isn't going to be happy with some of the paths this theoretically produces

https://github.com/pandas-dev/pandas/issues/22234
This is also failing on 1.5.x, can we backport or should we pin the ubuntu version? probably fine to backport? @meeseeksdev please backport to 1.5.x Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.5.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 5fe95c60c7b1db349a92e36b23816cfd4f2cc515
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #50117: Alternate Ubuntu CI fix'
```

4. Push to a named branch:

```
git push YOURFORK 1.5.x:auto-backport-of-pr-50117-on-1.5.x
```

5. Create a PR against branch 1.5.x, I would have named this PR:

> "Backport PR #50117 on branch 1.5.x (Alternate Ubuntu CI fix)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                