MAINT: Correct small cast issues on Windows
Use correct types to avoid downcasts on Windows

- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

avoids build warnings? Yes for some casts. Others are places where 54 but it's are being truncated to 32 bits. Would only ever show up on windows 64 in a frame with more then 2**32 rows. 