Revert "PERF: increase the minimum number of elements to use numexpr for ops from 1e4 to 1e6"
Reverts pandas-dev/pandas#40502

Troubleshooting the CI
Looks like this fixes all the builds except docbuild, which should be fixed by #40542 great merge when ready (and maybe open an issue about the perf change) This fixes the memory issue I noted on gitter:

> Anyone else seeing larger memory usage when running the tests? Each pytest worker is using close to 4 GiB on me, I recall this being 2 GiB not too long ago. Thanks @jbrockmendel  > Anyone else seeing larger memory usage when running the tests? Each pytest worker is using close to 4 GiB on me, I recall this being 2 GiB not too long ago.


Now that you mention it... about 1.3 GB at the end of collection, 6.04 by the time it hits 1%!  2GB matches my recollection of what ive seen recently pre-yesterday