BUG: don't cache pandas matplotlib converters (#27036)
- [ ] closes #27036 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

#27036 is caused by pandas converters replacing default converters in the cache when we register our  matplotlib converters.
Can you add a test for the bugfix How does this interact with https://github.com/pandas-dev/pandas/pull/27481? I believe this addresses the same issue as #27481, so I'm closing.