TST: revert parts of #32571
Should fix the broken CI builds.
Thanks. This shouldn't need to be backported.

Let's restart the 36-32bit and npdev builds a couple times to see if they show up. Well I don't see an easy way to restart just those builds so let's merge this when travis passes and see what happens.