Fix some little-endian assumptions in the tests
See previous [PR#46681](https://github.com/pandas-dev/pandas/pull/46681).

- [ ] closes #xxxx **N/A**
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature **N/A, fixes some existing test failures on s390x**
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions. **N/A**
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature. **N/A**

can you run the pre-commit checks please? Long line broken and force-pushed; pre-commit passes now. Thanks @musicinmybrain 