<p>I'm a little confused why my <code>rancher-agent</code> is no longer able to connect to the cluster server. This was working for me for a long time, but it appears to have broken on its own. DNS and networking confuses me.</p>
<p>My setup:</p>
<ul>
<li>OS: <code>Ubuntu 20.04.6 LTS</code></li>
<li>Docker: <code>Docker version 23.0.1</code></li>
<li>Rancher: <code>v2.6.5</code></li>
</ul>
<p>I have configured my cluster to run a single node, as <a href="https://ranchermanager.docs.rancher.com/v2.6/pages-for-subheaders/rancher-on-a-single-node-with-docker" rel="nofollow noreferrer">specified here</a>, and then I followed the <a href="https://ranchermanager.docs.rancher.com/v2.6/reference-guides/single-node-rancher-in-docker/advanced-options" rel="nofollow noreferrer">advanced setup instructions</a> to run <code>rancher/rancher</code> and <code>rancher/rancher-agent</code> on the same node.</p>
<h2>The issue</h2>
<p>Everything boots and runs. I can access all my applications in my cluster from <code>https://homelab.local</code> and everything loads and runs. My rancher admin UI boots on <code>https://homelab.local:8443/dashboard/home</code>. The issue is that I cannot manage the cluster at all.</p>
<p>I see these two errors under Cluster Management:
<code>Unsupported Docker version found [23.0.1] on host [192.168.0.75], supported versions are [1.13.x 17.03.x 17.06.x 17.09.x 18.06.x 18.09.x 19.03.x 20.10.x]</code><br />
and<br />
<code>[Disconnected] Cluster agent is not connected</code></p>
<p>So it appears that I have inadvertently upgraded Docker and this is breaking my cluster?</p>
<p>When I run kubectl get pods, I get some kind of cert error:</p>
<pre><code>kubectl get nodes
E0326 19:56:23.504726   70231 memcache.go:265] couldn't get current server API group list: Get &quot;https://localhost:8443/api?timeout=32s&quot;: x509: certificate signed by unknown authority
E0326 19:56:23.506701   70231 memcache.go:265] couldn't get current server API group list: Get &quot;https://localhost:8443/api?timeout=32s&quot;: x509: certificate signed by unknown authority
E0326 19:56:23.508357   70231 memcache.go:265] couldn't get current server API group list: Get &quot;https://localhost:8443/api?timeout=32s&quot;: x509: certificate signed by unknown authority
E0326 19:56:23.510425   70231 memcache.go:265] couldn't get current server API group list: Get &quot;https://localhost:8443/api?timeout=32s&quot;: x509: certificate signed by unknown authority
E0326 19:56:23.513743   70231 memcache.go:265] couldn't get current server API group list: Get &quot;https://localhost:8443/api?timeout=32s&quot;: x509: certificate signed by unknown authority
Unable to connect to the server: x509: certificate signed by unknown authority
</code></pre>
<p>How can I get my cluster back to a good state?</p>
<h2>Update</h2>
<p>I uninstalled the latest Docker with:<br />
<code>sudo apt-get remove docker-ce docker-ce-cli docker-ce-rootless-extras docker-compose-plugin docker-scan-plugin docker-buildx-plugin</code>
And installed Rancher's supported version like this:<br />
<code>curl https://releases.rancher.com/install-docker/20.10.sh | sh</code></p>
<p>This fixes the issue with the unsupported Docker version, but the rancher-agent image is still not booting up. When I look at the logs of the container, I see this:</p>
<pre><code>time=&quot;2023-03-27T03:20:59Z&quot; level=fatal msg=&quot;Certificate chain is not complete, please check if all needed intermediate certificates are included in the server certificate (in the correct order) and if the cacerts setting in Rancher either contains the correct CA certificate (in the case of using self signed certificates) or is empty (in the case of using a certificate signed by a recognized CA). Certificate information is displayed above. error: Get \&quot;https://192.168.0.75:8443\&quot;: x509: certificate signed by unknown authority&quot;
</code></pre>

RommelTJ
<p>As explained in this official  <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#tls-certificate-errors" rel="nofollow noreferrer">doc</a></p>
<blockquote>
<p><strong>The following error “Unable to connect to the server: x509:
certificate signed by unknown authority&quot; indicates a possible
certificate mismatch.</strong></p>
<p>When you run</p>
<pre><code>kubectl get pods
</code></pre>
<p>If you are getting this error Unable to connect to the server: x509:
certificate signed by unknown authority</p>
<p><strong>To resolve this error try below troubleshooting methods</strong>:</p>
<p>1)Verify that the $HOME/.kube/config file contains a valid
certificate, and regenerate a certificate if necessary. The
certificates in a kubeconfig file are base64 encoded. The base64
--decode command can be used to decode the certificate and openssl x509 -text -noout can be used for viewing the certificate information.</p>
<p>2)Unset the KUBECONFIG environment variable using:</p>
<pre><code>unset KUBECONFIG
</code></pre>
<p>Or set it to the default KUBECONFIG location:</p>
<pre><code>export KUBECONFIG=/etc/kubernetes/admin.conf
</code></pre>
<p>3)Another workaround is to overwrite the existing kubeconfig for the
&quot;admin&quot; user:</p>
<pre><code>    mv  $HOME/.kube $HOME/.kube.bak
    
    mkdir $HOME/.kube
    
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
</code></pre>
</blockquote>
<p>Refer to this official <a href="https://ranchermanager.docs.rancher.com/v2.5/getting-started/installation-and-upgrade/resources/custom-ca-root-certificates" rel="nofollow noreferrer">doc</a> for more information.</p>
