<p>I am using the <a href="https://github.com/kubernetes-incubator/client-python/" rel="noreferrer">kubernetes python client</a>.  In the event that kubernetes isn't available when my code starts up, I would like to retry the connection.  </p>

<p>When the client is unable to connect, it throws what appears to be a <code>urllib3.exceptions.MaxRetryError</code> exception, so I started with something like this:</p>

<pre><code>import time
import urllib3

import kubernetes

kubernetes.config.load_kube_config()
api = kubernetes.client.CoreV1Api()

while True:
    try:
        w = kubernetes.watch.Watch()
        for event in w.stream(api.list_pod_for_all_namespaces):
            print event
    except urllib3.exceptions.HTTPError:
        print('retrying in 1 second')
        time.sleep(1)
</code></pre>

<p>But that completely fails; it acts like there is no <code>except</code> statement and bails out with:</p>

<pre><code>urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.122.140', port=8443): Max retries exceeded with url: /api/v1/pods?watch=True (Caused by NewConnectionError('&lt;urllib3.connection.VerifiedHTTPSConnection object at 0x2743110&gt;: Failed to establish a new connection: [Errno 111] Connection refused',))
</code></pre>

<p>I thought maybe I didn't understand inheritance as well as I thought, so I replace the above with:</p>

<pre><code>except urllib3.exceptions.MaxRetryError:
    print('retrying in 1 second')
    time.sleep(1)
</code></pre>

<p>Which fails in the same way.  In an effort to figure out what was going on, I added a catch-all <code>except</code> and invoked pdb:</p>

<pre><code>except Exception as err:
    import pdb; pdb.set_trace()
</code></pre>

<p>And from the <code>pdb</code> prompt, we can see:</p>

<pre><code>(Pdb) type(err)
&lt;class 'urllib3.exceptions.MaxRetryError'&gt;
</code></pre>

<p>...which looks fine, as does the mro:</p>

<pre><code>(Pdb) import inspect
(Pdb) inspect.getmro(err.__class__)
(&lt;class 'urllib3.exceptions.MaxRetryError'&gt;, &lt;class 'urllib3.exceptions.RequestError'&gt;, &lt;class 'urllib3.exceptions.PoolError'&gt;, &lt;class 'urllib3.exceptions.HTTPError'&gt;, &lt;type 'exceptions.Exception'&gt;, &lt;type 'exceptions.BaseException'&gt;, &lt;type 'object'&gt;)
</code></pre>

<p>But despite all that:</p>

<pre><code>(Pdb) isinstance(err, urllib3.exceptions.MaxRetryError)
False
</code></pre>

<p>And all the paths look reasonable:</p>

<pre><code>(Pdb) urllib3.__file__
'/usr/lib/python2.7/site-packages/urllib3/__init__.pyc'
(Pdb) kubernetes.client.rest.urllib3.__file__
'/usr/lib/python2.7/site-packages/urllib3/__init__.pyc'
</code></pre>

<p>So...what the actual what is going on here?</p>

<p><strong>Update</strong></p>

<p>Here is the full stack trace:</p>

<pre><code>Traceback (most recent call last):
  File "testkube.py", line 13, in &lt;module&gt;
    for event in w.stream(api.list_pod_for_all_namespaces):
  File "/usr/lib/python2.7/site-packages/kubernetes/watch/watch.py", line 116, in stream
    resp = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14368, in list_pod_for_all_namespaces
    (data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14464, in list_pod_for_all_namespaces_with_http_info
    collection_formats=collection_formats)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
    _preload_content, _request_timeout)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 371, in request
    headers=headers)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 250, in GET
    query_params=query_params)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 223, in request
    headers=headers)
  File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request
    **urlopen_kw)
  File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
    **response_kw)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
    **response_kw)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
    **response_kw)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.122.140', port=8443): Max retries exceeded with url: /api/v1/pods?watch=True (Caused by NewConnectionError('&lt;urllib3.connection.VerifiedHTTPSConnection object at 0x3d16110&gt;: Failed to establish a new connection: [Errno 111] Connection refused',))
</code></pre>

<p>Your code has an indirect dependency on the <code>requests</code> package, and the <code>requests</code> package has a strange submodule called <code>requests.packages</code>. This used to contain copied source code from a number of dependencies, including <code>urllib3</code>, but they stopped doing that. They wanted to keep <code>requests.packages</code> around for backward compatibility, though, so now they're doing something weird.</p>

<p>Instead of <code>requests.packages</code> including a complete copy of the <code>urllib3</code> source code, it now imports <code>urllib3</code> and sets <code>sys.modules['requests.packages.urllib3'] = urllib3</code>. Depending on the <code>requests</code> version, it may set a number of other <code>sys.modules</code> entries, too; for example, as of requests <a href="https://github.com/requests/requests/blob/v2.18.4/requests/packages.py" rel="noreferrer">2.18.4</a>, the source code does</p>

<pre><code>for package in ('urllib3', 'idna', 'chardet'):
    locals()[package] = __import__(package)
    # This traversal is apparently necessary such that the identities are
    # preserved (requests.packages.urllib3.* is urllib3.*)
    for mod in list(sys.modules):
        if mod == package or mod.startswith(package + '.'):
            sys.modules['requests.packages.' + mod] = sys.modules[mod]
</code></pre>

<p>but in <a href="https://github.com/requests/requests/blob/v2.17.0/requests/packages.py" rel="noreferrer">2.17.0</a>, it does</p>

<pre><code>import urllib3
sys.modules['requests.packages.urllib3'] = urllib3

import idna
sys.modules['requests.packages.idna'] = idna

import chardet
sys.modules['requests.packages.chardet'] = chardet
</code></pre>

<p>This code interacts badly with submodules of the imported packages. If some code tries to do <code>import requests.packages.urllib3.exceptions</code> and Python doesn't find a <code>sys.modules['requests.packages.urllib3.exceptions']</code> entry, Python will <em>recreate</em> the <code>urllib3.exceptions</code> module and set <code>urllib3.exceptions</code> and <code>sys.modules['requests.packages.urllib3.exceptions']</code> to the new module (but it won't touch <code>sys.modules['urllib3.exceptions']</code>. This will generate new copies of the classes involved, causing your error.</p>

<p>A related problem with the same cause was <a href="https://github.com/requests/requests/issues/4102" rel="noreferrer">reported</a> back in May, leading to the new code shown in 2.18.4. 2.18.4 shouldn't cause the specific problems you're seeing, but it's still fragile, because if any submodules of <code>urllib3</code> aren't yet loaded at the time <code>requests.packages</code> screws with <code>sys.modules</code>, those submodules will exhibit the same problems you've seen today.</p>
