TST: use `with` where possible instead of manual `close`
- [ ] closes #xxxx (Replace xxxx with the Github issue number)
  - Nothing to close – just noticed these while working on #48922.
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
  - Only tests changed. 
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
  - Nothing changed.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.
  - Not done (not required)
Looks like there are some test failures from the CI

> would this need a whatsnew entry?

Nope @mroeschke Rebased (#48925, #48926 conflicted...) and CI passes now. Awesome! Thanks @akx. Object closing PRs are very much appreciated.