<p>I'm currently getting errors when trying to exec or get logs for my pods on my GKE cluster.</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl logs &lt;POD-NAME&gt;
Error from server: Get &quot;https://&lt;NODE-PRIVATE-IP&gt;:10250/containerLogs/default/&lt;POD-NAME&gt;/&lt;DEPLOYMENT-NAME&gt;&quot;: remote error: tls: internal error
</code></pre>
<pre class="lang-sh prettyprint-override"><code>$ kubectl exec -it &lt;POD-NAME&gt; -- sh
Error from server: error dialing backend: remote error: tls: internal error
</code></pre>
<p>One suspicious thing I found while troubleshooting is that all CSRs are getting denied...</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl get csr
NAME        AGE     SIGNERNAME                      REQUESTOR                 CONDITION
csr-79zkn   4m16s   kubernetes.io/kubelet-serving   system:node:&lt;NODE-NAME&gt;   Denied
csr-7b5sx   91m     kubernetes.io/kubelet-serving   system:node:&lt;NODE-NAME&gt;   Denied
csr-7fzjh   103m    kubernetes.io/kubelet-serving   system:node:&lt;NODE-NAME&gt;   Denied
csr-7gstl   19m     kubernetes.io/kubelet-serving   system:node:&lt;NODE-NAME&gt;   Denied
csr-7hrvm   11m     kubernetes.io/kubelet-serving   system:node:&lt;NODE-NAME&gt;   Denied
csr-7mn6h   87m     kubernetes.io/kubelet-serving   system:node:&lt;NODE-NAME&gt;   Denied
csr-7nd7h   4m57s   kubernetes.io/kubelet-serving   system:node:&lt;NODE-NAME&gt;   Denied
...
</code></pre>
<p>Any idea why this is happening ? Maybe a firewall issue ?</p>
<p>Thanks in advance !</p>
<h3>Update 1</h3>
<p>Here the same commands with a verbose output <code>--v=8</code>, <em>withtout the <code>goroutines</code> stack trace</em></p>
<p><code>$ kubectl logs --v=8 &lt;POD-NAME&gt;</code></p>
<pre><code>I0527 09:27:59.624843   10407 loader.go:375] Config loaded from file:  /home/kevin/.kube/config
I0527 09:27:59.628621   10407 round_trippers.go:420] GET https://&lt;PUBLIC-IP&gt;/api/v1/namespaces/default/pods/&lt;POD-NAME&gt;
I0527 09:27:59.628635   10407 round_trippers.go:427] Request Headers:
I0527 09:27:59.628644   10407 round_trippers.go:431]     Accept: application/json, */*
I0527 09:27:59.628649   10407 round_trippers.go:431]     User-Agent: kubectl/v1.19.3 (linux/amd64) kubernetes/1e11e4a
I0527 09:27:59.727411   10407 round_trippers.go:446] Response Status: 200 OK in 98 milliseconds
I0527 09:27:59.727461   10407 round_trippers.go:449] Response Headers:
I0527 09:27:59.727480   10407 round_trippers.go:452]     Audit-Id: ...
I0527 09:27:59.727496   10407 round_trippers.go:452]     Cache-Control: no-cache, private
I0527 09:27:59.727512   10407 round_trippers.go:452]     Content-Type: application/json
I0527 09:27:59.727528   10407 round_trippers.go:452]     Date: Thu, 27 May 2021 07:27:59 GMT
I0527 09:27:59.727756   10407 request.go:1097] Response Body: {&quot;kind&quot;:&quot;Pod&quot;,&quot;apiVersion&quot;:&quot;v1&quot;,&quot;metadata&quot;:{&quot;name&quot;:&quot;&lt;POD-NAME&gt;&quot;,&quot;generateName&quot;:&quot;&lt;POD-BASE-NAME&gt;&quot;,&quot;namespace&quot;:&quot;default&quot;,&quot;selfLink&quot;:&quot;/api/v1/namespaces/default/pods/&lt;POD-NAME&gt;&quot;,&quot;uid&quot;:&quot;...&quot;,&quot;resourceVersion&quot;:&quot;6764210&quot;,&quot;creationTimestamp&quot;:&quot;2021-05-19T10:33:28Z&quot;,&quot;labels&quot;:{&quot;app&quot;:&quot;&lt;NAME&gt;&quot;,&quot;pod-template-hash&quot;:&quot;...&quot;},&quot;ownerReferences&quot;:[{&quot;apiVersion&quot;:&quot;apps/v1&quot;,&quot;kind&quot;:&quot;ReplicaSet&quot;,&quot;name&quot;:&quot;&lt;POD-BASE-NAME&gt;&quot;,&quot;uid&quot;:&quot;...&quot;,&quot;controller&quot;:true,&quot;blockOwnerDeletion&quot;:true}],&quot;managedFields&quot;:[{&quot;manager&quot;:&quot;kube-controller-manager&quot;,&quot;operation&quot;:&quot;Update&quot;,&quot;apiVersion&quot;:&quot;v1&quot;,&quot;time&quot;:&quot;2021-05-19T10:33:28Z&quot;,&quot;fieldsType&quot;:&quot;FieldsV1&quot;,&quot;fieldsV1&quot;:{&quot;f:metadata&quot;:{&quot;f:generateName&quot;:{},&quot;f:labels&quot;:{&quot;.&quot;:{},&quot;f:app&quot;:{},&quot;f:pod-template-hash&quot;:{}},&quot;f:ownerReferences&quot;:{&quot;.&quot;:{},&quot;k:{\&quot;uid\&quot;:\&quot;...\&quot;}&quot;:{&quot;.&quot;:{},&quot;f:apiVersion&quot;:{},&quot;f:blockOwnerDeletion&quot;:{},&quot;f:controller&quot;:{},&quot;f:kind&quot;:{},&quot;f:name&quot;:{},&quot;f:uid&quot;:{}}}},&quot;f:spec&quot;:{&quot;f:c [truncated 3250 chars]
I0527 09:27:59.745985   10407 round_trippers.go:420] GET https://&lt;PUBLIC-IP&gt;/api/v1/namespaces/default/pods/&lt;POD-NAME&gt;/log
I0527 09:27:59.746035   10407 round_trippers.go:427] Request Headers:
I0527 09:27:59.746055   10407 round_trippers.go:431]     Accept: application/json, */*
I0527 09:27:59.746071   10407 round_trippers.go:431]     User-Agent: kubectl/v1.19.3 (linux/amd64) kubernetes/1e11e4a
I0527 09:27:59.800586   10407 round_trippers.go:446] Response Status: 500 Internal Server Error in 54 milliseconds
I0527 09:27:59.800638   10407 round_trippers.go:449] Response Headers:
I0527 09:27:59.800654   10407 round_trippers.go:452]     Audit-Id: ...
I0527 09:27:59.800668   10407 round_trippers.go:452]     Cache-Control: no-cache, private
I0527 09:27:59.800680   10407 round_trippers.go:452]     Content-Type: application/json
I0527 09:27:59.800693   10407 round_trippers.go:452]     Content-Length: 217
I0527 09:27:59.800712   10407 round_trippers.go:452]     Date: Thu, 27 May 2021 07:27:59 GMT
I0527 09:27:59.800772   10407 request.go:1097] Response Body: {&quot;kind&quot;:&quot;Status&quot;,&quot;apiVersion&quot;:&quot;v1&quot;,&quot;metadata&quot;:{},&quot;status&quot;:&quot;Failure&quot;,&quot;message&quot;:&quot;Get \&quot;https://10.156.0.8:10250/containerLogs/default/&lt;POD-NAME&gt;/&lt;SERVICE-NAME&gt;\&quot;: remote error: tls: internal error&quot;,&quot;code&quot;:500}
I0527 09:27:59.801848   10407 helpers.go:216] server response object: [{
  &quot;metadata&quot;: {},
  &quot;status&quot;: &quot;Failure&quot;,
  &quot;message&quot;: &quot;Get \&quot;https://10.156.0.8:10250/containerLogs/default/&lt;POD-NAME&gt;/&lt;SERVICE-NAME&gt;\&quot;: remote error: tls: internal error&quot;,
  &quot;code&quot;: 500
}]
F0527 09:27:59.801944   10407 helpers.go:115] Error from server: Get &quot;https://10.156.0.8:10250/containerLogs/default/&lt;POD-NAME&gt;/&lt;SERVICE-NAME&gt;&quot;: remote error: tls: internal error
</code></pre>
<p><code>kubectl exec --v=8 -it &lt;POD-NAME&gt; -- sh</code></p>
<pre><code>I0527 09:44:48.673774   11157 loader.go:375] Config loaded from file:  /home/kevin/.kube/config
I0527 09:44:48.678514   11157 round_trippers.go:420] GET https://&lt;PUBLIC-IP&gt;/api/v1/namespaces/default/pods/&lt;POD-NAME&gt;
I0527 09:44:48.678528   11157 round_trippers.go:427] Request Headers:
I0527 09:44:48.678535   11157 round_trippers.go:431]     Accept: application/json, */*
I0527 09:44:48.678543   11157 round_trippers.go:431]     User-Agent: kubectl/v1.19.3 (linux/amd64) kubernetes/1e11e4a
I0527 09:44:48.795864   11157 round_trippers.go:446] Response Status: 200 OK in 117 milliseconds
I0527 09:44:48.795920   11157 round_trippers.go:449] Response Headers:
I0527 09:44:48.795963   11157 round_trippers.go:452]     Audit-Id: ...
I0527 09:44:48.795995   11157 round_trippers.go:452]     Cache-Control: no-cache, private
I0527 09:44:48.796019   11157 round_trippers.go:452]     Content-Type: application/json
I0527 09:44:48.796037   11157 round_trippers.go:452]     Date: Thu, 27 May 2021 07:44:48 GMT
I0527 09:44:48.796644   11157 request.go:1097] Response Body: {&quot;kind&quot;:&quot;Pod&quot;,&quot;apiVersion&quot;:&quot;v1&quot;,&quot;metadata&quot;:{&quot;name&quot;:&quot;&lt;POD-NAME&gt;&quot;,&quot;generateName&quot;:&quot;&quot;,&quot;namespace&quot;:&quot;default&quot;,&quot;selfLink&quot;:&quot;/api/v1/namespaces/default/pods/&lt;POD-NAME&gt;&quot;,&quot;uid&quot;:&quot;&quot;,&quot;resourceVersion&quot;:&quot;6764210&quot;,&quot;creationTimestamp&quot;:&quot;2021-05-19T10:33:28Z&quot;,&quot;labels&quot;:{&quot;app&quot;:&quot;...&quot;,&quot;pod-template-hash&quot;:&quot;...&quot;},&quot;ownerReferences&quot;:[{&quot;apiVersion&quot;:&quot;apps/v1&quot;,&quot;kind&quot;:&quot;ReplicaSet&quot;,&quot;name&quot;:&quot;&lt;POD-BASE-NAME&gt;&quot;,&quot;uid&quot;:&quot;...&quot;,&quot;controller&quot;:true,&quot;blockOwnerDeletion&quot;:true}],&quot;managedFields&quot;:[{&quot;manager&quot;:&quot;kube-controller-manager&quot;,&quot;operation&quot;:&quot;Update&quot;,&quot;apiVersion&quot;:&quot;v1&quot;,&quot;time&quot;:&quot;2021-05-19T10:33:28Z&quot;,&quot;fieldsType&quot;:&quot;FieldsV1&quot;,&quot;fieldsV1&quot;:{&quot;f:metadata&quot;:{&quot;f:generateName&quot;:{},&quot;f:labels&quot;:{&quot;.&quot;:{},&quot;f:app&quot;:{},&quot;f:pod-template-hash&quot;:{}},&quot;f:ownerReferences&quot;:{&quot;.&quot;:{},&quot;k:{\&quot;uid\&quot;:\&quot;...\&quot;}&quot;:{&quot;.&quot;:{},&quot;f:apiVersion&quot;:{},&quot;f:blockOwnerDeletion&quot;:{},&quot;f:controller&quot;:{},&quot;f:kind&quot;:{},&quot;f:name&quot;:{},&quot;f:uid&quot;:{}}}},&quot;f:spec&quot;:{&quot;f:c [truncated 3250 chars]
I0527 09:44:48.814315   11157 round_trippers.go:420] POST https://&lt;PUBLIC-IP&gt;/api/v1/namespaces/default/pods/&lt;POD-NAME&gt;/exec?command=sh&amp;container=&lt;SERVICE-NAME&gt;&amp;stdin=true&amp;stdout=true&amp;tty=true
I0527 09:44:48.814372   11157 round_trippers.go:427] Request Headers:
I0527 09:44:48.814391   11157 round_trippers.go:431]     User-Agent: kubectl/v1.19.3 (linux/amd64) kubernetes/1e11e4a
I0527 09:44:48.814406   11157 round_trippers.go:431]     X-Stream-Protocol-Version: v4.channel.k8s.io
I0527 09:44:48.814420   11157 round_trippers.go:431]     X-Stream-Protocol-Version: v3.channel.k8s.io
I0527 09:44:48.814445   11157 round_trippers.go:431]     X-Stream-Protocol-Version: v2.channel.k8s.io
I0527 09:44:48.814471   11157 round_trippers.go:431]     X-Stream-Protocol-Version: channel.k8s.io
I0527 09:44:48.913928   11157 round_trippers.go:446] Response Status: 500 Internal Server Error in 99 milliseconds
I0527 09:44:48.913977   11157 round_trippers.go:449] Response Headers:
I0527 09:44:48.914005   11157 round_trippers.go:452]     Audit-Id: ...
I0527 09:44:48.914029   11157 round_trippers.go:452]     Cache-Control: no-cache, private
I0527 09:44:48.914054   11157 round_trippers.go:452]     Content-Type: application/json
I0527 09:44:48.914077   11157 round_trippers.go:452]     Date: Thu, 27 May 2021 07:44:48 GMT
I0527 09:44:48.914099   11157 round_trippers.go:452]     Content-Length: 149
I0527 09:44:48.915741   11157 helpers.go:216] server response object: [{
  &quot;metadata&quot;: {},
  &quot;status&quot;: &quot;Failure&quot;,
  &quot;message&quot;: &quot;error dialing backend: remote error: tls: internal error&quot;,
  &quot;code&quot;: 500
}]
F0527 09:44:48.915837   11157 helpers.go:115] Error from server: error dialing backend: remote error: tls: internal error
</code></pre>
<h3>Update 2</h3>
<p>After connecting to one of the GKE worker nodes and checking <code>kubelet</code> logs I found these wired lines</p>
<pre><code>May 27 09:30:11 gke-&lt;CLUSTER-NAME&gt;-default-pool-&lt;NODE-UID&gt; kubelet[1272]: I0527 09:30:11.271022    1272 log.go:181] http: TLS handshake error from 10.156.0.9:54672: no serving certificate available for the kubelet
May 27 09:30:11 gke-&lt;CLUSTER-NAME&gt;-default-pool-&lt;NODE-UID&gt; kubelet[1272]: I0527 09:30:11.305628    1272 log.go:181] http: TLS handshake error from 10.156.0.9:54674: no serving certificate available for the kubelet
May 27 09:30:12 gke-&lt;CLUSTER-NAME&gt;-default-pool-&lt;NODE-UID&gt; kubelet[1272]: I0527 09:30:12.067998    1272 log.go:181] http: TLS handshake error from 10.156.0.11:57610: no serving certificate available for the kubelet
May 27 09:30:14 gke-&lt;CLUSTER-NAME&gt;-default-pool-&lt;NODE-UID&gt; kubelet[1272]: I0527 09:30:14.144826    1272 certificate_manager.go:412] Rotating certificates
May 27 09:30:14 gke-&lt;CLUSTER-NAME&gt;-default-pool-&lt;NODE-UID&gt; kubelet[1272]: I0527 09:30:14.154322    1272 reflector.go:207] Starting reflector *v1.CertificateSigningRequest (0s) from k8s.io/client-go/tools/watch/informerwatcher.go:146
May 27 09:30:14 gke-&lt;CLUSTER-NAME&gt;-default-pool-&lt;NODE-UID&gt; kubelet[1272]: I0527 09:30:14.448976    1272 reflector.go:213] Stopping reflector *v1.CertificateSigningRequest (0s) from k8s.io/client-go/tools/watch/informerwatcher.go:146
May 27 09:30:14 gke-&lt;CLUSTER-NAME&gt;-default-pool-&lt;NODE-UID&gt; kubelet[1272]: E0527 09:30:14.449045    1272 certificate_manager.go:454] certificate request was not signed: cannot watch on the certificate signing request: certificate signing request is denied, reason: AutoDenied, message:
</code></pre>
<h3>Update 3</h3>
<p>I've updated the cluster version from <code>1.19.9-gke.1400</code> to <code>1.19.9-gke.1900</code>. Didn't solved the problem...</p>
<p>Made a <code>Credentials Rotation</code> on the cluster. But didn't solved as well...</p>
<h1>Final</h1>
<p>After trying lot of changes in the cluster :</p>
<ul>
<li>Restarting kubelet on nodes</li>
<li>Restarting nodes</li>
<li>Upscaling/Downscaling node pool size</li>
<li>Upgrading cluster version</li>
<li>Rotating cluster certificates</li>
</ul>
<p>Even creating a new cluster (on the same project, with same VPC, etc) didn't solve the issue...</p>
<p><strong>This problem might be related to changes made on Firewall rules.</strong></p>
<p>Only solution found, creating a new GKE cluster in a new GCP project and migrating workflow using Velero.</p>

<p>Assume this answer may help ,</p>
<p>*This issue is due to <strong>pending</strong> Certificate Sigining Requests  for  nodes made by kubelet running in each node *</p>
<p>to check the csr pending for nodes</p>
<pre><code>kubectl get csr  --sort-by=.metadata.creationTimestamp
</code></pre>
<p>then approve the csr for each node</p>
<pre><code>kubectl certificate  approve  &lt;csr-id&gt;
</code></pre>
<p>Further details ,refer this section in k8s documentation</p>
<blockquote>
<p><a href="https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#kubelet-serving-certs" rel="noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#kubelet-serving-certs</a></p>
<p>One known limitation is that the CSRs (Certificate Signing Requests)
for these certificates cannot be automatically approved by the default
signer in the kube-controller-manager - kubernetes.io/kubelet-serving.
This will require action from the user or a third party controller</p>
</blockquote>
<p>.</p>
