<p>We are using elasticsearch/kibana instead of gcp for logging (based on what is described <a href="https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch" rel="nofollow noreferrer">here</a>).</p>

<p>To have fluentd-elsticsearch pod's launched we've set <code>LOGGING_DESTINATION=elasticsearch</code> and <code>ENABLE_NODE_LOGGING="true"</code> in the "Compute Instance Template" -> "Custom metadata" -> "kube-env".</p>

<p>While this works fine when done manually it gets overwritten with every <code>gcloud container clusters upgrade</code> as a new Instance Template with defaults (<code>LOGGING_DESTINATION=gcp</code> ...) is created.</p>

<p>My question is: How do I persist this kind of configuration for GKE/GCE?</p>

<p>I thought about adding a <em>k8s-user-startup-script</em> but that's also defined in the Instance Template and therefore is overwritten by <code>gcloud container clusters upgrade</code>.
I've also tried to add a <em>k8s-user-startup-script</em> to the project metadata but that is not taken into account.  </p>

<p>//EDIT<br>
Current workaround (without recreating Instance Template and Instances) for manually switching back to elasticsearch is:</p>

<pre><code>for node in $(kubectl get nodes -o name | cut -f2 -d/); do
    gcloud compute ssh $node \
      --command="sudo cp -a /srv/salt/fluentd-es/fluentd-es.yaml /etc/kubernetes/manifests/; sudo rm /etc/kubernetes/manifests/fluentd-gcp.yaml";
done
</code></pre>

<p>kubelet will pick that up, kill fluentd-gcp and start fluentd-es.</p>

<p>//EDIT #2
Now running a "startup-script" DaemonSet for this:</p>

<pre><code>kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: startup-script
  namespace: kube-system
  labels:
    app: startup-script
spec:
  template:
    metadata:
      labels:
        app: startup-script
    spec:
      hostPID: true
      containers:
        - name: startup-script
          image: gcr.io/google-containers/startup-script:v1
          securityContext:
            privileged: true
          env:
          - name: STARTUP_SCRIPT
            value: |
              #! /bin/bash

              set -o errexit
              set -o pipefail
              set -o nounset

              # Replace Google-Cloud-Logging with EFK
              if [[ ! -f /etc/kubernetes/manifests/fluentd-es.yaml ]]; then
                if [[ -f /home/kubernetes/kube-manifests/kubernetes/fluentd-es.yaml ]]; then
                  # GCI images
                  cp -a /home/kubernetes/kube-manifests/kubernetes/fluentd-es.yaml /etc/kubernetes/manifests/
                elif [[ -f /srv/salt/fluentd-es/fluentd-es.yaml ]]; then
                  # Debian based GKE images
                  cp -a /srv/salt/fluentd-es/fluentd-es.yaml /etc/kubernetes/manifests/
                fi
                test -f /etc/kubernetes/manifests/fluentd-es.yaml &amp;&amp; rm /etc/kubernetes/manifests/fluentd-gcp.yaml
              fi
</code></pre>

<p>There isn't a fully supported way to reconfigure the kube-env in GKE. As you've found, you can hack the instance template, but this isn't guaranteed to work across upgrades.</p>

<p>An alternative is to create your cluster without gcp logging enabled and then create a DaemonSet that places a fluentd-elasticsearch pod on each of your nodes. Using this technique you don't need to write a (brittle) startup script or rely on the fact that the built-in startup script happens to work when setting <code>LOGGING_DESTINATION=elasticsearch</code> (which may break across upgrades even if it wasn't getting overwritten). </p>
