CI: Remove 32 bit testing
- [ ] closes #28842 (Replace xxxx with the GitHub issue number)
- [ ] closes #44453 (Replace xxxx with the GitHub issue number)
- [ ] closes #46360 (Replace xxxx with the GitHub issue number)

With the meson build PR, we dropped windows 32 bit wheels. Additionally, other pydata libraries are also starting to remove 32 bit wheel support https://discuss.python.org/t/dropping-32-bit-packages/5476/9. Since we don't ship these wheels anymore, I am not sure if we should continue to test this specifically. 


This was also previously discussed in e.g. https://github.com/pandas-dev/pandas/issues/15889 and the consensus seemed to be that it's worth supporting 32-bit builds in source, even though wheels would not be published. Especially:
> Moreover, my very general and superficial analysis of issues like https://github.com/pandas-dev/pandas/issues/24613 (or - now speaking out of experience - https://github.com/pandas-dev/pandas/issues/19439 ) is that they actually help us fix mistakes in the code, regardless of the use of 32 bits.

Seems like the challenge of *windows* 32-bit *release wheels* shouldn't reflect on the challenge of running CI on linux. Ah okay. I suppose we can keep this around and support those who install from source on 32 bit platforms. Closing In https://github.com/pandas-dev/pandas/issues/44453, I also [say](https://github.com/pandas-dev/pandas/issues/44453#issuecomment-968468140):
>Apologies if I'm misunderstanding here, but it looks like Numpy isn't dropping 32-bit support https://github.com/numpy/numpy/issues/20277#issuecomment-957529088. It looks like they're only dropping wheel generation because of infra constraints but will continue to test.
>
>I'm interested because in Gentoo, we still support a number of 32-bit architectures (arm, x86 being the two main ones), and Pandas ends up in the depgraph often unavoidably if we want to run test suites on programs.
>
>Is the plan for Pandas to drop support entirely for 32-bit or to just stop producing wheels?

Numpy continues to have this position, per https://discuss.python.org/t/dropping-32-bit-packages/5476/16. > Ah okay. I suppose we can keep this around and support those who install from source on 32 bit platforms. Closing

Thank you! It is really appreciated, I promise.