CI: enable parallel testing on arm64 build #36719
- [x] closes #36719


It took about 19min to setup the environment and cache... make sure it's actually using the workers in setup.py (-j)

and u can use 4 looks like it completed (with some failure)

mark the longest tests with arm_slow and try more workers  I did a simple test on my Travis account. 

<details>

```yml
os: linux

dist: bionic

language: python

python: 3.8

arch:
  - arm64
  - arm64-graviton2

addons:
  apt:
    packages:
      - libsnappy-dev

branches:
  only:
    - main

before_install:
  - |
    git clone https://github.com/pandas-dev/pandas.git
    cd pandas
install:
  - python -c 'import os,sys,fcntl; flags = fcntl.fcntl(sys.stdout, fcntl.F_GETFL); fcntl.fcntl(sys.stdout, fcntl.F_SETFL, flags&~os.O_NONBLOCK);'
  - python -m pip install --no-deps -U pip wheel setuptools
  - python -m pip install cython numpy python-dateutil pytz pytest pytest-xdist hypothesis
  - python setup.py build_ext -j4
  - python -m pip install -e . --no-build-isolation --no-use-pep517

script:
  - pytest -n 4 -m 'not slow and not network and not clipboard' pandas --junitxml=test-data.xml
```

</details>

It only took about 7 min to setup and 21 min to run the test. Let's see if using 4 cores would cut the test time to 20 min range. let start by limiting the directories it is running 

we do this for the windows builds (for reference) > let start by limiting the directories it is running
> 
> we do this for the windows builds (for reference)

I did some tests locally. The config that affects the runtime the most is the distribution algorithm. We were using `loadfile`, which group tests by file. It's extremely slow on arm machine. I changed it to the default value `no`, which assigns tests to workers one by one. Now, the test only took 1131.13s. wow this looks great

ok can u open an issue for the failing arm tests themselves, xfail them and then make this build a required one > wow this looks great
> 
> ok can u open an issue for the failing arm tests themselves, xfail them and then make this build a required one

I xfailed those tests. But they still failed on CI. I tested them with py38 and numpydev on M1 mbp. The rolling test passed without xfail. But the other two tests passed with xfail.  hmm do we need to disable the cache for arm? /azp run <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp> <samp>
Commenter does not have sufficient privileges for PR 38905 in repo pandas-dev/pandas<br>

</samp> The test failure in py37 macOS build is caused by the --dist=no. note that travis is currently not running at all :-< working on a credit issue there let's close for now as travis does not look like its coming back