title: Why did we get auto-upgraded to 14.7-gke.14? Why is the latest 1.14.8-gke.2 not working at all…
body: <p>Seems like our master got auto-upgraded this morning without prior warning from an earlier <code>1.14.x</code> version to <code>1.14.7-gke.14</code> but we’ve disabled auto-upgrade before so it’s completely odd especially given that <code>1.14.7-gke.14</code> is breaking workload identity which quite a few of us in the GKE world rely on. </p>
  
  <p>Moreover, I’m seeing a new version in the list <code>1.14.8-gke.2</code> which at the time of writing this question was not even mentioned in <a href="https://cloud.google.com/kubernetes-engine/docs/release-notes" rel="nofollow noreferrer">the GKE release notes</a> at all. Our cluster was created before release channels became a thing so AFAIK we're not enrolled in any of those.</p>
  
  <p>Wondering if someone pulled the release trigger prematurely (again as this is not the first time).</p>
  
  <p>Since there was no option to go back to a stable version where workload identity was working fine, we went ahead and upgraded to <code>1.14.8-gke.2</code> in a hope it would fix the problem. This was a clear gamble that did not pay off at all because it resulted in a colossal clusterfuck where some services did not come up at all due to all kinds of container and networking issues. Specifically:</p>
  
  <ol>
  <li><p>kube-dns's container prometheus-to-sd is throwing the following error: <code>http://169.254.169.254/computeMetadata/v1/project/project-id: dial tcp 169.254.169.254:80: getsockopt: connection refused</code></p></li>
  <li><p>readiness and liveness probes are failing due to some sort of internal networking issue for some of our services</p></li>
  </ol>
  
  <p>Now on to the main question. What would you guys suggest? Recreate a cluster and go back to v1.13.x and also enroll in a stable release channel? What other options do we have? Would like to get some insight into what the heck is going on here.</p>
  
  <p>EDIT: we're currently hosted in <code>europe-west1-d</code>, it's a zonal cluster that comprises 1 standard pool w/ 3 auto-scaled nodes and 1 preemptible pool for dynamic workloads. </p>
  
  <p>EDIT2: just noticed in the release notes from Oct 30 that 1.14.6-gke.2 was going to be wound down, I must have missed it before. Either way the question is still why there's an undocumented version and why it's not working + if anyone knows when workload identity will work again in the regular channel? This forces us to downgrade to a 1.13.x</p>
  
  <p>EDIT3: turns out kube-dns is using a 1.15.4 image version on a 1.14.x cluster, this almost looks like a broken release, when i downgrade it works for a brief moment until the cluster controller decided to revert the change so we're going to recreate the cluster</p>
  
  <p>EDIT4: after a few hours it turns out that <code>cos_containerd</code> image was the main culprit, switching the image helped resolve all issues it seems, we'll keep monitoring</p>