title: Issue upgrading calico-node in kubeadm cluster
body: <p>I am going to <a href="https://docs.projectcalico.org/v2.0/getting-started/kubernetes/upgrade" rel="nofollow noreferrer">upgrade Calico node and cni as per this link for "Upgrading Components Individually"</a></p>
  
  <p>The directions are very clear (I will cordon each node and do the step for the <code>calico/cni</code> and <code>calico/node</code>), but I am not sure what is meant by </p>
  
  <p><code>Update the image in your process management to reference the new version</code></p>
  
  <p>wrt to upgrading the <code>calico/node</code> container.</p>
  
  <p>Otherwise, I see no other issues wrt the directions. Our environment is a k8s kubeadm cluster.</p>
  
  <p>I suppose the real question is: where do I tell k8s to use the newer version of the <code>calico/node</code> image?</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>To answer the above:</p>
  
  <p>I just did a <code>kubectl delete -f</code> on both <code>calico.yaml</code> and <code>rbac-kdd.yaml</code> and then did a <code>kubectl create -f</code> on the newest version of these files. </p>
  
  <p>Everything appears now to be at version 3.3.2, but I am getting this error now on all the calico-node pods:</p>
  
  <p><code>Warning  Unhealthy  84s (x181 over 31m)  kubelet, thalia4  Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with &lt;node IP addresses here</code></p>
  
  <p>I ran <code>calicoctl nodd status</code> and got</p>
  
  <pre><code>Calico process is running.
  
  IPv4 BGP status
  +---------------+-------------------+-------+----------+--------------------------------+
  | PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |              INFO              |
  +---------------+-------------------+-------+----------+--------------------------------+
  | 134.x.x.163 | node-to-node mesh | start | 02:36:29 | Connect                        |
  | 134.x.x.164 | node-to-node mesh | start | 02:36:29 | Connect                        |
  | 134.x.x.165 | node-to-node mesh | start | 02:36:29 | Connect                        |
  | 134.x.x.168 | node-to-node mesh | start | 02:36:29 | Active Socket: Host is         |
  |             |                   |       |          | unreachable                    |
  +---------------+-------------------+-------+----------+--------------------------------+
  
  IPv6 BGP status
  No IPv6 peers found.
  </code></pre>
  
  <p>I would assume 134.x.x.168 being unreachable is why I am getting the above health check warning. </p>
  
  <p>Not exactly sure what to do though. This node is available in the k8s cluster (this is node <code>thalia4</code>):</p>
  
  <pre><code>[gms@thalia0 calico]$ kubectl get nodes
  NAME                  STATUS   ROLES    AGE   VERSION
  thalia0               Ready    master   87d   v1.13.1
  thalia1               Ready    &lt;none&gt;   48d   v1.13.1
  thalia2               Ready    &lt;none&gt;   30d   v1.13.1
  thalia3               Ready    &lt;none&gt;   87d   v1.13.1
  thalia4               Ready    &lt;none&gt;   48d   v1.13.1
  </code></pre>
  
  <p><strong><em>EDIT 2</em></strong></p>
  
  <p><code>calicoctl node status</code> on thalia4 gave</p>
  
  <pre><code>[sudo] password for gms:
  Calico process is running.
  
  IPv4 BGP status
  +---------------+-------------------+-------+----------+---------+
  | PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |  INFO   |
  +---------------+-------------------+-------+----------+---------+
  | 134.xx.xx.162 | node-to-node mesh | start | 02:36:29 | Connect |
  | 134.xx.xx.163 | node-to-node mesh | start | 02:36:29 | Connect |
  | 134.xx.xx.164 | node-to-node mesh | start | 02:36:29 | Connect |
  | 134.xx.xx.165 | node-to-node mesh | start | 02:36:29 | Connect |
  +---------------+-------------------+-------+----------+---------+
  </code></pre>
  
  <p>while <code>kubectl describe node thalia4</code> gave</p>
  
  <pre><code>Name:               thalia4.domain
  Roles:              &lt;none&gt;
  Labels:             beta.kubernetes.io/arch=amd64
                      beta.kubernetes.io/os=linux
                      dns=dns4
                      kubernetes.io/hostname=thalia4
                      node_name=thalia4
  Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                      node.alpha.kubernetes.io/ttl: 0
                      projectcalico.org/IPv4Address: 134.xx.xx.168/26
                      volumes.kubernetes.io/controller-managed-attach-detach: true
  CreationTimestamp:  Mon, 03 Dec 2018 14:17:07 -0600
  Taints:             &lt;none&gt;
  Unschedulable:      false
  Conditions:
    Type             Status    LastHeartbeatTime                 LastTransitionTime                Reason                       Message
    ----             ------    -----------------                 ------------------                ------                       -------
    OutOfDisk        Unknown   Fri, 21 Dec 2018 11:58:38 -0600   Sat, 12 Jan 2019 16:44:10 -0600   NodeStatusUnknown            Kubelet stopped posting node status.
    MemoryPressure   False     Mon, 21 Jan 2019 20:54:38 -0600   Sat, 12 Jan 2019 16:50:18 -0600   KubeletHasSufficientMemory   kubelet has sufficient memory available
    DiskPressure     False     Mon, 21 Jan 2019 20:54:38 -0600   Sat, 12 Jan 2019 16:50:18 -0600   KubeletHasNoDiskPressure     kubelet has no disk pressure
    PIDPressure      False     Mon, 21 Jan 2019 20:54:38 -0600   Sat, 12 Jan 2019 16:50:18 -0600   KubeletHasSufficientPID      kubelet has sufficient PID available
    Ready            True      Mon, 21 Jan 2019 20:54:38 -0600   Sun, 20 Jan 2019 20:27:10 -0600   KubeletReady                 kubelet is posting ready status
  Addresses:
    InternalIP:  134.xx.xx.168
    Hostname:    thalia4
  Capacity:
   cpu:                4
   ephemeral-storage:  6878Mi
   hugepages-1Gi:      0
   hugepages-2Mi:      0
   memory:             8009268Ki
   pods:               110
  Allocatable:
   cpu:                4
   ephemeral-storage:  6490895145
   hugepages-1Gi:      0
   hugepages-2Mi:      0
   memory:             7906868Ki
   pods:               110
  System Info:
   Machine ID:                 c011569a40b740a88a672a5cc526b3ba
   System UUID:                42093037-F27E-CA90-01E1-3B253813B904
   Boot ID:                    ffa5170e-da2b-4c09-bd8a-032ce9fca2ee
   Kernel Version:             3.10.0-957.1.3.el7.x86_64
   OS Image:                   Red Hat Enterprise Linux
   Operating System:           linux
   Architecture:               amd64
   Container Runtime Version:  docker://1.13.1
   Kubelet Version:            v1.13.1
   Kube-Proxy Version:         v1.13.1
  PodCIDR:                     192.168.4.0/24
  Non-terminated Pods:         (3 in total)
    Namespace                  Name                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
    ---------                  ----                        ------------  ----------  ---------------  -------------  ---
    kube-system                calico-node-8xqbs           250m (6%)     0 (0%)      0 (0%)           0 (0%)         24h
    kube-system                coredns-786f4c87c8-sbks2    100m (2%)     0 (0%)      70Mi (0%)        170Mi (2%)     47h
    kube-system                kube-proxy-zp4fk            0 (0%)        0 (0%)      0 (0%)           0 (0%)         31d
  Allocated resources:
    (Total limits may be over 100 percent, i.e., overcommitted.)
    Resource           Requests   Limits
    --------           --------   ------
    cpu                350m (8%)  0 (0%)
    memory             70Mi (0%)  170Mi (2%)
    ephemeral-storage  0 (0%)     0 (0%)
  Events:              &lt;none&gt;
  </code></pre>
  
  <p>I'm thinking this is a firewall problem, but I was told on the Slack channel that "If you're not using host endpoints then we don't mess with your host's connectivity.  It sounds like you've got something blocking port 179 on that host." </p>
  
  <p>Not sure where that would be? The iptables rules look the same across all nodes.</p>