<p>Yesterday, <a href="https://github.com/naftulikay/katyperry" rel="noreferrer">I built a full-featured example</a> which uses Terraform to create a network and a GKE cluster in Google Cloud Platform. The whole thing runs in Vagrant on a CentOS 7 VM and installs both <code>gcloud</code>, <code>kubectl</code>, and <code>helm</code>. I also <a href="https://github.com/naftulikay/katyperry/blob/master/doc/SPINNAKER.md" rel="noreferrer">extended the example</a> to use Helm to install Spinnaker.</p>

<p>The GKE cluster is called <code>gke-test-1</code>. In my documentation I documented getting <code>kubectl</code> setup:</p>

<pre><code>gcloud container clusters get-credentials --region=us-west1 gke-test-1
</code></pre>

<p>After this, I was able to use various <code>kubectl</code> commands to <code>get nodes</code>, <code>get pods</code>, <code>get services</code>, and <code>get deployments</code>, as well as all other cluster management commands. I was able to also use Helm to install Tiller and ultimately deploy Spinnaker.</p>

<p>However, today, the same process doesn't work for me. I spun up the network, subnet, GKE cluster, and the node pool, and whenever I try to use commands to get various resoures, I get this response:</p>

<pre><code>[vagrant@katyperry vagrant]$ kubectl get nodes
No resources found.
Error from server (NotAcceptable): unknown (get nodes)
[vagrant@katyperry vagrant]$ kubectl get pods
No resources found.
Error from server (NotAcceptable): unknown (get pods)
[vagrant@katyperry vagrant]$ kubectl get services
No resources found.
Error from server (NotAcceptable): unknown (get services)
[vagrant@katyperry vagrant]$ kubectl get deployments
No resources found.
Error from server (NotAcceptable): unknown (get deployments.extensions)
</code></pre>

<p>Interestingly enough, it seems that some command do work:</p>

<pre><code>[vagrant@katyperry vagrant]$ kubectl describe nodes | head
Name:               gke-gke-test-1-default-253fb645-scq8
Roles:              &lt;none&gt;
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/fluentd-ds-ready=true
                    beta.kubernetes.io/instance-type=n1-standard-4
                    beta.kubernetes.io/os=linux
                    cloud.google.com/gke-nodepool=default
                    failure-domain.beta.kubernetes.io/region=us-west1
                    failure-domain.beta.kubernetes.io/zone=us-west1-b
                    kubernetes.io/hostname=gke-gke-test-1-default-253fb645-scq8
</code></pre>

<p>When I open a shell in Google Cloud console, after running the same login command, I'm able to use <code>kubectl</code> to do all of the above:</p>

<pre><code>naftuli_kay@naftuli-test:~$ gcloud beta container clusters get-credentials gke-test-1 --region us-west1 --project naftuli-test
Fetching cluster endpoint and auth data.
kubeconfig entry generated for gke-test-1.
naftuli_kay@naftuli-test:~$ kubectl get pods
No resources found.
naftuli_kay@naftuli-test:~$ kubectl get nodes
NAME                                   STATUS    ROLES     AGE       VERSION
gke-gke-test-1-default-253fb645-scq8   Ready     &lt;none&gt;    40m       v1.8.10-gke.0
gke-gke-test-1-default-253fb645-tfns   Ready     &lt;none&gt;    40m       v1.8.10-gke.0
gke-gke-test-1-default-8bf306fc-n8jz   Ready     &lt;none&gt;    40m       v1.8.10-gke.0
gke-gke-test-1-default-8bf306fc-r0sq   Ready     &lt;none&gt;    40m       v1.8.10-gke.0
gke-gke-test-1-default-aecb57ba-85p4   Ready     &lt;none&gt;    40m       v1.8.10-gke.0
gke-gke-test-1-default-aecb57ba-n7n3   Ready     &lt;none&gt;    40m       v1.8.10-gke.0
naftuli_kay@naftuli-test:~$ kubectl get services
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.0.64.1    &lt;none&gt;        443/TCP   43m
naftuli_kay@naftuli-test:~$ kubectl get deployments
No resources found.
</code></pre>

<p>The only difference I can see is the difference between the <code>kubectl</code> version; Vagrant has the latest version, 1.11.0, and the Google Cloud console has 1.9.7.</p>

<p>I will attempt to downgrade.</p>

<p>Is this a known issue and what, if anything, can I do to work around it?</p>

<hr>

<p><strong>EDIT:</strong> This is reproducible and I can't find a way to prevent it from recurring. I tore down all of my infrastructure and then stood it up again. The Terraform is available <a href="https://github.com/naftulikay/katyperry" rel="noreferrer">here</a>. </p>

<p>After provisioning the resources, I waited until the cluster reported being healthy:</p>

<pre><code>[vagrant@katyperry vagrant]$ gcloud container clusters describe \
    --region=us-west1 gke-test-1 | grep -oP '(?&lt;=^status:\s).*'
RUNNING
</code></pre>

<p>I then setup my login credentials:</p>

<pre><code>[vagrant@katyperry vagrant]$ gcloud container clusters get-credentials \
    --region=us-west1 gke-test-1
</code></pre>

<p>I again attempted to get nodes:</p>

<pre><code>[vagrant@katyperry vagrant]$ kubectl get nodes
No resources found.
Error from server (NotAcceptable): unknown (get nodes)
</code></pre>

<p>The cluster appears green in the Google Cloud dashboard:</p>

<p><a href="https://i.stack.imgur.com/AiDR7.png" rel="noreferrer"><img src="https://i.stack.imgur.com/AiDR7.png" alt="enter image description here"></a></p>

<p>Apparently, this is a reproducible problem, as I'm able to recreate it using the same Terraform and commands. </p>

<p>After successfully reproducing the issue multiple times by destroying and recreating all the infrastructure, I found <a href="https://gitlab.com/charts/gitlab/merge_requests/227#note_68683418" rel="noreferrer">some arcane post on GitLab</a> that mentions <a href="https://github.com/kubernetes/kubernetes/issues/61943#issuecomment-377586904" rel="noreferrer">a Kubernetes GitHub issue</a> that seems to indicate:</p>

<blockquote>
  <p>...in order to maintain compatibility with 1.8.x servers (which are within the supported version skew of +/- one version)</p>
</blockquote>

<p>Emphasis on the "+/- one version."</p>

<p><a href="https://github.com/naftulikay/katyperry/pull/1" rel="noreferrer">Upgrading the masters and the workers to Kubernetes 1.10</a> seems to entirely have addressed the issue, as I can now list nodes and pods with impunity:</p>

<pre><code>[vagrant@katyperry vagrant]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.4-gke.2", GitCommit:"eb2e43842aaa21d6f0bb65d6adf5a84bbdc62eaf", GitTreeState:"clean", BuildDate:"2018-06-15T21:48:39Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}
[vagrant@katyperry vagrant]$ kubectl get nodes
NAME                                   STATUS    ROLES     AGE       VERSION
gke-gke-test-1-default-5989a78d-dpk9   Ready     &lt;none&gt;    42s       v1.10.4-gke.2
gke-gke-test-1-default-5989a78d-kh9b   Ready     &lt;none&gt;    58s       v1.10.4-gke.2
gke-gke-test-1-default-653ba633-091s   Ready     &lt;none&gt;    46s       v1.10.4-gke.2
gke-gke-test-1-default-653ba633-4zqq   Ready     &lt;none&gt;    46s       v1.10.4-gke.2
gke-gke-test-1-default-848661e8-cv53   Ready     &lt;none&gt;    53s       v1.10.4-gke.2
gke-gke-test-1-default-848661e8-vfr6   Ready     &lt;none&gt;    52s       v1.10.4-gke.2
</code></pre>

<p>It appears that Google Cloud Platform's cloud shell pins to <code>kubectl</code> 1.9, which is within the version gap supported by the ideas expressed above.</p>

<p>Thankfully, the Kubernetes RHEL repository has a bunch of versions to choose from so it's possible to pin:</p>

<pre><code>[vagrant@katyperry gke]$ yum --showduplicates list kubectl
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.usc.edu
 * epel: sjc.edge.kernel.org
 * extras: mirror.sjc02.svwh.net
 * updates: mirror.linuxfix.com
Installed Packages
kubectl.x86_64                                          1.11.0-0                                          @kubernetes
Available Packages
kubectl.x86_64                                          1.5.4-0                                           kubernetes
kubectl.x86_64                                          1.6.0-0                                           kubernetes
kubectl.x86_64                                          1.6.1-0                                           kubernetes
kubectl.x86_64                                          1.6.2-0                                           kubernetes
kubectl.x86_64                                          1.6.3-0                                           kubernetes
kubectl.x86_64                                          1.6.4-0                                           kubernetes
kubectl.x86_64                                          1.6.5-0                                           kubernetes
kubectl.x86_64                                          1.6.6-0                                           kubernetes
kubectl.x86_64                                          1.6.7-0                                           kubernetes
kubectl.x86_64                                          1.6.8-0                                           kubernetes
kubectl.x86_64                                          1.6.9-0                                           kubernetes
kubectl.x86_64                                          1.6.10-0                                          kubernetes
kubectl.x86_64                                          1.6.11-0                                          kubernetes
kubectl.x86_64                                          1.6.12-0                                          kubernetes
kubectl.x86_64                                          1.6.13-0                                          kubernetes
kubectl.x86_64                                          1.7.0-0                                           kubernetes
kubectl.x86_64                                          1.7.1-0                                           kubernetes
kubectl.x86_64                                          1.7.2-0                                           kubernetes
kubectl.x86_64                                          1.7.3-1                                           kubernetes
kubectl.x86_64                                          1.7.4-0                                           kubernetes
kubectl.x86_64                                          1.7.5-0                                           kubernetes
kubectl.x86_64                                          1.7.6-1                                           kubernetes
kubectl.x86_64                                          1.7.7-1                                           kubernetes
kubectl.x86_64                                          1.7.8-1                                           kubernetes
kubectl.x86_64                                          1.7.9-0                                           kubernetes
kubectl.x86_64                                          1.7.10-0                                          kubernetes
kubectl.x86_64                                          1.7.11-0                                          kubernetes
kubectl.x86_64                                          1.7.14-0                                          kubernetes
kubectl.x86_64                                          1.7.15-0                                          kubernetes
kubectl.x86_64                                          1.7.16-0                                          kubernetes
kubectl.x86_64                                          1.8.0-0                                           kubernetes
kubectl.x86_64                                          1.8.1-0                                           kubernetes
kubectl.x86_64                                          1.8.2-0                                           kubernetes
kubectl.x86_64                                          1.8.3-0                                           kubernetes
kubectl.x86_64                                          1.8.4-0                                           kubernetes
kubectl.x86_64                                          1.8.5-0                                           kubernetes
kubectl.x86_64                                          1.8.6-0                                           kubernetes
kubectl.x86_64                                          1.8.7-0                                           kubernetes
kubectl.x86_64                                          1.8.8-0                                           kubernetes
kubectl.x86_64                                          1.8.9-0                                           kubernetes
kubectl.x86_64                                          1.8.10-0                                          kubernetes
kubectl.x86_64                                          1.8.11-0                                          kubernetes
kubectl.x86_64                                          1.8.12-0                                          kubernetes
kubectl.x86_64                                          1.8.13-0                                          kubernetes
kubectl.x86_64                                          1.8.14-0                                          kubernetes
kubectl.x86_64                                          1.9.0-0                                           kubernetes
kubectl.x86_64                                          1.9.1-0                                           kubernetes
kubectl.x86_64                                          1.9.2-0                                           kubernetes
kubectl.x86_64                                          1.9.3-0                                           kubernetes
kubectl.x86_64                                          1.9.4-0                                           kubernetes
kubectl.x86_64                                          1.9.5-0                                           kubernetes
kubectl.x86_64                                          1.9.6-0                                           kubernetes
kubectl.x86_64                                          1.9.7-0                                           kubernetes
kubectl.x86_64                                          1.9.8-0                                           kubernetes
kubectl.x86_64                                          1.10.0-0                                          kubernetes
kubectl.x86_64                                          1.10.1-0                                          kubernetes
kubectl.x86_64                                          1.10.2-0                                          kubernetes
kubectl.x86_64                                          1.10.3-0                                          kubernetes
kubectl.x86_64                                          1.10.4-0                                          kubernetes
kubectl.x86_64                                          1.10.5-0                                          google-cloud-sdk
kubectl.x86_64                                          1.10.5-0                                          kubernetes
kubectl.x86_64                                          1.11.0-0                                          kubernetes
</code></pre>

<hr>

<p><strong>EDIT:</strong> I have found the <a href="https://github.com/kubernetes/kubernetes/pull/61419" rel="noreferrer">actual pull request that mentions this incompatibility</a>. I have also found <a href="https://kubernetes.io/docs/imported/release/notes/" rel="noreferrer">buried in the release notes</a> the following information:</p>

<blockquote>
  <p>kubectl: This client version requires the <strong><code>apps/v1</code></strong> API, so it will not work against a cluster version older than v1.9.0. Note that kubectl only guarantees compatibility with clusters that are +/- [one] minor version away. </p>
</blockquote>

<h2>TL;DR</h2>

<p>This entire problem was an incompatibility between <code>kubectl</code> 1.11 and Kubernetes 1.8.</p>
