<p><strong>Versions:</strong> Kubernetes <code>v1.9.2</code> running in GCE (kube-up.sh)</p>

<p>Container-Optimized OS <code>10032.88.0</code></p>

<hr>

<p><strong>Symptom:</strong> Our COS nodes show the following in their <code>google-ip-forwarding-daemon.service</code> logs:</p>

<pre><code>Mar 07 13:43:28 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.188', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.188', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.154.243.135'].
Mar 07 13:43:45 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.188', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.198.28.188'].
Mar 07 13:44:01 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.171.xxx'].
Mar 07 13:44:17 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'130.211.120.8'].
Mar 07 13:44:37 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.198.52.xxx'].
Mar 07 13:44:53 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.17.xx'].
Mar 07 13:45:10 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.155.181.xx'].
Mar 07 13:45:26 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.255.xxx'].
Mar 07 13:45:43 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.166.xxx'].
Mar 07 13:45:59 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] by adding None and removing [u'146.148.61.xx'].
Mar 07 13:46:19 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.197.104.xxx'].
Mar 07 13:46:41 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.198.172.xxx'].
Mar 07 13:46:58 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.198.162.xxx'].
Mar 07 13:47:14 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.155.131.xxx'].
Mar 07 13:47:34 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'130.211.238.xxx'] by adding None and removing [u'104.198.131.xxx'].
Mar 07 13:47:53 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'130.211.238.xxx'] by adding None and removing [u'104.154.72.xxx'].
Mar 07 13:48:10 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'130.211.238.xxx'] to [u'130.211.238.xxx'] by adding None and removing [u'104.154.20.xx'].
Mar 07 13:48:26 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'130.211.238.xxx'] to None by adding None and removing [u'130.211.238.xxx'].
</code></pre>

<p>When this happens, running PODs typically continue running yet sometimes become Unknown and are rescheduled. Any newly scheduled pods are stuck in ContainerCreating with these errors:</p>

<p><code>Failed create pod sandbox.</code> &amp; <code>Error response from daemon: network none not found</code></p>

<p>We haven't found a fix to this symptom other than rebooting the instance.</p>

<p>There are too many versions between Kubernetes 1.2 and 1.9.2, why are you trying to create a very old version and then upgrade it? I followed the same Kubernetes doc and got a cluster running 1.9.3 without issues. You should try the same and verify if the issue you described persist.</p>

<p>Additionally, the image <a href="https://cloud.google.com/container-optimized-os/docs/release-notes#cos-stable-63-10032-88-0" rel="nofollow noreferrer">cos-stable-63-10032-88-0</a> is not the latest release. You should also try running with <a href="https://cloud.google.com/container-optimized-os/docs/release-notes#cos-stable-64-10176-62-0" rel="nofollow noreferrer">cos-stable-64-10176-62-0</a> instead.</p>
