TST: Refactor some slow tests
Mainly refactoring tests by reducing the data size not associated with the bug or refactoring the test to more directly test the issue
Looks like tests are passing. Going to merge and happy to address followups 