CI: Temporarily build pandas with N_JOBS=1 to avoid flakiness
xref: #47305

Does this affect the CI speed? Looks like the build step goes from ~4-6 min to ~8-10 min. IMO it's an okay trade off to prevent flakiness.

Plus caching these builds might be in the works soon https://github.com/pandas-dev/pandas/issues/46643  Works for me The ImportError hasn't show up when only using 1 worker after a few merges of main, so I think this temporarily solves the flakiness. Merging. 