<p>I'm trying to get elasticsearch running on Kubernetes 1.16 with Helm 3 on GKE. I'm aware that both 1.16 and 3 aren't supported yet. I want to prepare a PR to make it compatible. I'm using the helm charts from <a href="https://github.com/elastic/helm-charts" rel="nofollow noreferrer">https://github.com/elastic/helm-charts</a>.</p>

<p>If I use the original chart 7.6.1 the pod creation fails due to <code>create Pod elasticsearch-master-0 in StatefulSet elasticsearch-master failed error: pods "elasticsearch-master-0" is forbidden: unable to validate against any pod security policy: [spec.volumes[1]: Invalid value: "projected": projected volumes are not allowed to be used]</code>. Therefore I created the following patch:</p>

<pre><code>diff --git a/elasticsearch/values.yaml b/elasticsearch/values.yaml
index 053c020..fd9c37b 100755
--- a/elasticsearch/values.yaml
+++ b/elasticsearch/values.yaml
@@ -107,6 +107,7 @@ podSecurityPolicy:
       - secret
       - configMap
       - persistentVolumeClaim
+      - projected

 persistence:
   enabled: true
</code></pre>

<p>With this patch on master/d9ccb5a and tag 7.6.1 (tried both) the pods quickly get into unhealthy state due to <code>failed to resolve host [elasticsearch-master-headless]</code> caused by a <code>java.net.UnknownHostException: elasticsearch-master-headless</code>.</p>

<p>I don't understand why the name resolution doesn't work as there's no change introduced in 1.16 which changes name resolution with Kubernetes names afaik. If I try to ping <code>elasticsearch-master-headless</code> from a shell in the pod started with <code>kubectl exec</code>, I can't reach it neither.</p>

<p>I tried to contact the nameserver in <code>/etc/resolv.conf</code> with <code>telnet</code> because it allows specifying a specific port:</p>

<pre><code>[elasticsearch@elasticsearch-master-1 ~]$ cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local us-central1-a.c.myproject.internal c.myproject.internal google.internal
nameserver 10.23.240.10
options ndots:5
[elasticsearch@elasticsearch-master-1 ~]$ telnet 10.23.240.10
Trying 10.23.240.10...
^C
[elasticsearch@elasticsearch-master-1 ~]$ telnet 10.23.240.10 53
Trying 10.23.240.10...
telnet: connect to address 10.23.240.10: Connection refused
</code></pre>

<p>I obfuscated the project ID with myproject.</p>

<p>The patch is already proposed to be merged upstream together with other changes at <a href="https://github.com/elastic/helm-charts/pull/496" rel="nofollow noreferrer">https://github.com/elastic/helm-charts/pull/496</a>.</p>

Kalle Richter
<p>This is caused by the pod <code>kube-dns</code> crashing due to</p>

<pre><code>F0315 20:01:02.464839 1 server.go:61] Failed to create a kubernetes client: open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied
</code></pre>

<p>Since Kubernetes 1.16 is only available in the rapid channel of GKE and it's a system pod, I consider this a bug.</p>

<p>I'll update this answer if I find the energy to file a bug.</p>
