Correctly re-instate Matplotlib converters
Fixes https://github.com/pandas-dev/pandas/issues/27479. Converters should only be cached to be re-instated if they are original, and not pandas converters.

I'm struggling to write a test for this; I think it requires a clean state with no pandas imports, so one can check what was in the units registry before pandas was imported. I think this means a new test file with no pandas imports at the top, but maybe someone has a better idea?
Thanks for this!

> I'm struggling to write a test for this; 

Testing these is quite hard (we accidentally broke the warning for a release and didn't notice).

https://github.com/pandas-dev/pandas/blob/9bab81e013af33084c48b54dc2a6ef188acfd225/pandas/tests/plotting/test_converter.py#L31 works reasonably well I think. The idea is to run the check in a subprocess. A release note in whatsnew/0.25.1.rst would be nice to have. I think there's a "plotting" subsection. # [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27481?src=pr&el=h1) Report
> Merging [#27481](https://codecov.io/gh/pandas-dev/pandas/pull/27481?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/9bab81e013af33084c48b54dc2a6ef188acfd225?src=pr&el=desc) will **increase** coverage by `49.17%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/27481/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/27481?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #27481       +/-   ##
===========================================
+ Coverage   42.44%   91.62%   +49.17%     
===========================================
  Files         184      184               
  Lines       50389    50388        -1     
===========================================
+ Hits        21389    46167    +24778     
+ Misses      29000     4221    -24779
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `91.62% <100%> (?)` | |
| #single | `?` | |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/27481?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/plotting/\_matplotlib/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/27481/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19tYXRwbG90bGliL2NvbnZlcnRlci5weQ==) | `63.64% <100%> (+49.94%)` | :arrow_up: |
| [pandas/core/computation/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/27481/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vcHl0YWJsZXMucHk=) | `63.71% <0%> (-26.22%)` | :arrow_down: |
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/27481/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `65.55% <0%> (-23.83%)` | :arrow_down: |
| [pandas/io/clipboard/clipboards.py](https://codecov.io/gh/pandas-dev/pandas/pull/27481/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NsaXBib2FyZC9jbGlwYm9hcmRzLnB5) | `31.88% <0%> (-2.9%)` | :arrow_down: |
| [pandas/util/\_test\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/27481/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX3Rlc3RfZGVjb3JhdG9ycy5weQ==) | `93.84% <0%> (+4.61%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/27481/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (+5.55%)` | :arrow_up: |
| [pandas/\_config/localization.py](https://codecov.io/gh/pandas-dev/pandas/pull/27481/diff?src=pr&el=tree#diff-cGFuZGFzL19jb25maWcvbG9jYWxpemF0aW9uLnB5) | `86% <0%> (+6%)` | :arrow_up: |
| [pandas/core/dtypes/inference.py](https://codecov.io/gh/pandas-dev/pandas/pull/27481/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2luZmVyZW5jZS5weQ==) | `100% <0%> (+13.33%)` | :arrow_up: |
| [pandas/compat/numpy/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/27481/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9udW1weS9fX2luaXRfXy5weQ==) | `93.1% <0%> (+13.79%)` | :arrow_up: |
| [pandas/core/config\_init.py](https://codecov.io/gh/pandas-dev/pandas/pull/27481/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnX2luaXQucHk=) | `96.2% <0%> (+13.92%)` | :arrow_up: |
| ... and [138 more](https://codecov.io/gh/pandas-dev/pandas/pull/27481/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27481?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27481?src=pr&el=footer). Last update [9bab81e...9589b8f](https://codecov.io/gh/pandas-dev/pandas/pull/27481?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
 Thanks for the test tip, the new test fails before the changes in this PR. I'm away for the next week or so without a computer, so if anyone wants this in quicker than that feel free to force push to my branch. Is there anything else (apart from a rebase) I need to do on this?
 Did you merge master before pushing? We've seen similar CI failures, but I thought we fixed them all on master. Yeah, I should have rebased on to master, I've done another rebase to see if that fixes the tests. Merged master again to see if we can get the CI passing. Thanks @dstansby. Sorry about the CI issues.