Fix #54391: Use memcpy() for unaligned loads
- [x] closes #54391
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

@jonashaag 
@WillAyd I've coalesced all the changes together now Thanks @SoapGentoo 