BLD/CI fix arm64 build #36397
- [x] closes #36397 and 
closes #34149 and
closes #30041


I would put a marker on most of these tests so we skip on arm64 (as too slow), could use arm_slow or something (though also ok to mark the ones >40s as slow)

```
-- Docs: https://docs.pytest.org/en/stable/warnings.html
---- generated xml file: /home/travis/build/pandas-dev/pandas/test-data.xml ----
============================= slowest 30 durations =============================
64.26s call     pandas/tests/test_sorting.py::TestSorting::test_int64_overflow_moar
48.15s call     pandas/tests/groupby/transform/test_transform.py::test_cython_transform_frame[shift-args3-<lambda>]
47.99s call     pandas/tests/indexing/test_chaining_and_caching.py::TestChaining::test_detect_chained_assignment
44.52s call     pandas/tests/arithmetic/test_datetime64.py::TestDatetime64Arithmetic::test_dt64arr_add_timedeltalike_scalar[tzutc()-Timedelta-DataFrame]
42.54s call     pandas/tests/indexing/multiindex/test_chaining_and_caching.py::test_indexer_caching
38.96s call     pandas/tests/groupby/test_groupby_dropna.py::test_groupby_dropna_datetime_like_data[None-unique_nulls_fixture22-True-values0-datetime10-datetime20]
29.91s call     pandas/tests/frame/test_constructors.py::TestDataFrameConstructors::test_constructor_range_dtype[category]
29.43s call     pandas/tests/indexes/test_base.py::TestIndex::test_engine_reference_cycle
27.33s call     pandas/tests/tseries/offsets/test_offsets_properties.py::test_on_offset_implementations
26.79s call     pandas/tests/indexes/multi/test_duplicates.py::test_duplicated_large[first]
25.48s call     pandas/tests/indexes/multi/test_integrity.py::test_hash_collisions
25.16s call     pandas/tests/tseries/offsets/test_ticks.py::test_tick_equality[Nano]
23.06s call     pandas/tests/indexes/timedeltas/test_timedelta.py::TestTimedeltaIndex::test_engine_reference_cycle
22.52s call     pandas/tests/indexes/period/test_period.py::TestPeriodIndex::test_engine_reference_cycle
22.31s call     pandas/tests/indexes/categorical/test_category.py::TestCategoricalIndex::test_engine_reference_cycle
21.55s call     pandas/tests/indexes/test_base.py::TestMixedIntIndex::test_engine_reference_cycle
21.32s call     pandas/tests/indexes/datetimes/test_datetimelike.py::TestDatetimeIndex::test_engine_reference_cycle
21.02s call     pandas/tests/indexes/ranges/test_range.py::TestRangeIndex::test_engine_reference_cycle
20.21s call     pandas/tests/io/sas/test_sas7bdat.py::TestSAS7BDAT::test_iterator_loop
19.90s call     pandas/tests/io/sas/test_xport.py::TestXport::test1_basic
19.71s call     pandas/tests/groupby/transform/test_transform.py::test_cython_transform_frame[shift-args2-<lambda>]
18.88s call     pandas/tests/io/parser/test_parse_dates.py::test_bad_date_parse[python--True]
14.11s call     pandas/tests/io/sas/test_sas7bdat.py::TestSAS7BDAT::test_from_iterator
12.83s call     pandas/tests/io/parser/test_common.py::test_chunks_have_consistent_numerical_type[python]
11.45s call     pandas/tests/groupby/test_function.py::test_builtins_apply[keys1-min]
11.22s call     pandas/tests/io/parser/test_c_parser_only.py::test_grow_boundary_at_cap[c_low]
11.20s call     pandas/tests/generic/test_frame.py::TestDataFrame::test_head_tail[interval]
11.10s call     pandas/tests/io/parser/test_c_parser_only.py::test_grow_boundary_at_cap[c_high]
9.70s call     pandas/tests/groupby/test_function.py::test_builtins_apply[keys1-max]
9.38s call     pandas/tests/io/parser/test_common.py::test_warn_if_chunks_have_mismatched_type[python]
= 72115 passed, 4426 skipped, 1143 xfailed, 6 xpassed, 16 warnings in 1531.93s (0:25:31) =
``` retriggered this to see if the good perf repeats cc @TomAugspurger @simonjayhawkins 

doing for 1.2 until really think we need it for 1.1.3 (its really just a ci patch) 2nd run wasn' bad 36 min > doing for 1.2 until really think we need it for 1.1.3 (its really just a ci patch)

no strong preference. I think should back port unless we are really sure that 1.1.3 is the last bug fix release in the 1.1. series > I think should back port unless we are really sure that 1.1.3 is the last bug fix release in the 1.1. series

fair enough. thanks @VirosaLi 

this will likely be a manual backport as well :-> xref  #34131