<p>I can't create Persistent volume claim in Kubernetes (Rancher 2.3).</p>

<p>My storage class use VMware cloud provider 'vSphere Storage for Kubernetes' provided by Rancher</p>

<p>In rancher web interface, the Event show errors like:</p>

<pre><code>(combined from similar events): Failed to provision volume with StorageClass "t-a-g1000-t6m-e0fd": Post https://vsphere.exemple.com:443/sdk: dial tcp: lookup vsphere.exemple.com on [::1]:53: read udp [::1]:51136-&gt;[::1]:53: read: connection refused
</code></pre>

<p>I get the same error on my Kubernetes Master:</p>

<pre><code>docker logs kube-controller-manager
</code></pre>

Franklin Piat
<p>For some reason, the DNS resolver of the <code>kube-controller-manager</code> pod on kubernet master was empty:</p>

<pre><code>docker exec  kube-controller-manager cat /etc/resolv.conf
# Generated by dhcpcd
# /etc/resolv.conf.head can replace this line
# /etc/resolv.conf.tail can replace this line
</code></pre>

<p>Since the host server <code>resolv.conf</code>was correct, I simply restarted the container:</p>

<pre><code>docker restart kube-controller-manager
</code></pre>

<p>(an alternative ugly way would have been to edit the resolv.conf manually using <code>docker restart kube-controller-manager</code>, then do run the appropriate <code>echo XXX &gt;&gt; /etc/resolv.conf</code> ... bad idea  )</p>

<hr>

<p>Some other containers may have a similar issue on this node. This is a hacky way to identify and restart those containers:</p>

<pre><code>cd  cd /var/lib/docker/containers
ls -1 "$(grep nameserver -L */resolv.conf)" | sed -e 's#/.*##' 
0c10e1374644cc262c8186e28787f53e02051cc75c1f943678d7aeaa00e5d450
70fd116282643406b72d9d782857bb7ec76dd85dc8a7c0a83dc7ab0e90d30966
841def818a8b4df06a0d30b0b7a66b75a3b554fb5feffe78846130cdfeb39899
ae356e26f1bf8fafe530d57d8c68a022a0ee0e13b4e177d3ad6d4e808d1b36da
d593735a2f6d96bcab3addafcfe3d44b6d839d9f3775449247bdb8801e2e1692
d9b0dfaa270d3f50169fb1aed064ca7a594583b9f71a111f653c71d704daf391
</code></pre>

<p>Restart affected containers:</p>

<pre><code>cd /var/lib/docker/containers ; ls -1 $(grep nameserver -L */resolv.conf) | sed -e 's#/.*##' | xargs -n1 -r docker restart
</code></pre>
