BUG: Fix implementation of temp_setattr
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Doesn't fix any user-facing bug (at least, that I'm aware of), but the current implementation will not restore the original attribute if an exception is raised within the context manager.
thanks @rhshadrach 