<p>I have a cluster set up in Google Kubernetes Engine (GKE), with preemptible instances, TPU support, and 1 container per node.</p>
<p>About twice per container per day I get this error calling e.g. <code>tf.io.gfile.glob(...)</code>:</p>
<blockquote>
<p>tensorflow.python.framework.errors_impl.FailedPreconditionError: Error executing an HTTP request: libcurl code 6 meaning 'Couldn't resolve host name', error details: Couldn't resolve host 'www.googleapis.com' when reading gs://bucket/dir/dir</p>
</blockquote>
<p>It only happens in GKE, not when running directly on a compute VM in Google Cloud.</p>
<p>Is this something I should just expect in GKE and handle (e.g. back off and retry), or is there some kind of networking config issue or other underlying problem that I can do something about?</p>

<p>Since your cluster has only preemptible instances, the system pods will also restart every 24 hours (at least), including <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/" rel="nofollow noreferrer"><code>kube-dns</code></a>.<br />
You have a couple of options:</p>
<ol>
<li>Create another node pool (Of-course, this node pool doesn't have to include TPU nodes, you can try and use <code>n1-standard-1</code>, or even cheaper instances) in that GKE cluster to host the critical pods (e.g. <code>kube-dns</code>), and add a <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector" rel="nofollow noreferrer"><code>nodeSelector</code></a> field with that node pool to the critical deployments. For example:</li>
</ol>
<pre><code>nodeSelector:
        cloud.google.com/gke-nodepool: critical-node-pool
</code></pre>
<ol start="2">
<li>The number of <code>kube-dns</code> replicas is determined by <code>kube-dns-autoscaler</code> deployment. Increasing the number of <code>kube-dns</code> pods can help you make your system more resilient to node preemption. You can increase by editing the <code>kube-dns-autoscaler</code> ConfigMap in <code>kube-system</code> namespace.<br />
Edit the ConfigMap with <code>kubectl edit configmap dns-autoscaler --namespace=kube-system</code>. Look for this line: <code>linear: '{&quot;coresPerReplica&quot;:256,&quot;min&quot;:1,&quot;nodesPerReplica&quot;:16}'</code>.<br />
As explained <a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-horizontal-autoscaling/#tuning-autoscaling-parameters" rel="nofollow noreferrer">here</a>, the &quot;min&quot; field indicates the minimal number of DNS bakends. You can increase the minimum number of replicas, or tweak the other parameters the satisfy your needs.<br />
If you use the Auto-Scaling, don't set the minimum number of <code>kube-dns</code> replicas to an higher number than your minimal node count in that cluster, as it'll keep the nodes up just to due to deployment of <code>kube-dns</code>, and with no relation to your application workload.</li>
</ol>
