CI: troubleshoot segfault
I can't reproduce the CI failures locally, so just rolling back a few commits to narrow it down.
Looking at https://github.com/pandas-dev/pandas/issues/31922#issuecomment-585999883 it seems the test reverted in https://github.com/pandas-dev/pandas/pull/36511/commits/4679970b2d64d3c55150e0953d04a15d8ae23a48 wasn't supposed to run across all builds? > wasn't supposed to run across all builds?

Looks like, yah hmm this is failing older builds? ok so merging this and will re-open #31922 as this is supposed to run only on numpy master? (are the older build too old)?