<p>We have a 5-node cluster that was moved behind our corporate firewall/proxy server.</p>

<p>As per the directions here: <a href="https://medium.com/@gargankur74/setting-up-standalone-kubernetes-cluster-behind-corporate-proxy-on-ubuntu-16-04-1f2aaa5a848" rel="nofollow noreferrer">setting-up-standalone-kubernetes-cluster-behind-corporate-proxy</a></p>

<p>I set the proxy server environment variables using:</p>

<pre><code>export http_proxy=http://proxy-host:proxy-port/
export HTTP_PROXY=$http_proxy
export https_proxy=$http_proxy
export HTTPS_PROXY=$http_proxy
printf -v lan '%s,' localip_of_machine
printf -v pool '%s,' 192.168.0.{1..253}
printf -v service '%s,' 10.96.0.{1..253}
export no_proxy="${lan%,},${service%,},${pool%,},127.0.0.1";
export NO_PROXY=$no_proxy
</code></pre>

<p>Now everything in our cluster works internally. However, when I try to create a pod that pulls down an image from the outside, the pod is stuck on <code>ContainerCreating</code>, e.g.,</p>

<pre><code>[gms@thalia0 ~]$ kubectl apply -f https://k8s.io/examples/admin/dns/busybox.yaml
pod/busybox created
</code></pre>

<p>is stuck here:</p>

<pre><code>[gms@thalia0 ~]$ kubectl get pods
NAME                            READY   STATUS              RESTARTS   AGE
busybox                         0/1     ContainerCreating   0          17m
</code></pre>

<p>I assume this is due to the host/domain that the image is being pulled from not being in our corporate proxy rules. We do have rules for</p>

<pre><code>k8s.io
kubernetes.io
docker.io
docker.com
</code></pre>

<p>so, I'm not sure what other hosts/domains need to be added.</p>

<p>I did a describe pods for busybox and see reference to <code>node.kubernetes.io</code> (I am putting in a domain-wide exception for <code>*.kubernetes.io</code> which will hopefully suffice).</p>

<p>This is what I get from <code>kubectl describe pods busybox</code>:</p>

<pre><code>Volumes:
  default-token-2kfbw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-2kfbw
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age   From                          Message
  ----     ------                  ----  ----                          -------
  Normal   Scheduled               73s   default-scheduler             Successfully assigned default/busybox to thalia3.ahc.umn.edu
  Warning  FailedCreatePodSandBox  10s   kubelet, thalia3.ahc.umn.edu  Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "6af48c5dadf6937f9747943603a3951bfaf25fe1e714cb0b0cbd4ff2d59aa918" network for pod "busybox": NetworkPlugin cni failed to set up pod "busybox_default" network: error getting ClusterInformation: Get https://[10.96.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 10.96.0.1:443: i/o timeout, failed to clean up sandbox container "6af48c5dadf6937f9747943603a3951bfaf25fe1e714cb0b0cbd4ff2d59aa918" network for pod "busybox": NetworkPlugin cni failed to teardown pod "busybox_default" network: error getting ClusterInformation: Get https://[10.96.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 10.96.0.1:443: i/o timeout]
  Normal   SandboxChanged          10s   kubelet, thalia3.ahc.umn.edu  Pod sandbox changed, it will be killed and re-created.
</code></pre>

<p>I would assume the calico error is due to this:</p>

<pre><code>Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                     node.kubernetes.io/unreachable:NoExecute for 300s
</code></pre>

<p>The <code>calico</code> and <code>coredns</code> pods seem to have similar errors reaching <code>node.kubernetes.io</code>, so I would assume this is due to our server not being able to pull down the new images on a restart.</p>

<p>It looks like you are misunderstanding a few Kubernetes concepts that I'd like to help clarify here. References to <code>node.kubernetes.io</code> is not an attempt make any network calls to that domain. It is simply the convention that Kubernetes uses to specify string keys. So if you ever have to apply labels, annotations, or tolerations, you would define your own keys like <code>subdomain.domain.tld/some-key</code>.</p>

<p>As for the Calico issue that you are experiencing, it looks like the error:</p>

<pre><code>network: error getting ClusterInformation: Get https://[10.96.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 10.96.0.1:443: i/o timeout]
</code></pre>

<p>is our culprit here. <code>10.96.0.1</code> is the IP address used to refer to the Kubernetes API server within pods. It seems like the <code>calico/node</code> pod running on your node is failing to reach the API server. Could you more context around how you set up Calico? Do you know what version of Calico you are running?</p>

<p>The fact that your <code>calico/node</code> instance is trying to access the <code>crd.projectcalico.org/v1/clusterinformations</code> resource tells me that it is using the Kubernetes datastore for its backend. Are you sure you're not trying to run Calico in Etcd mode?</p>
