<p>This is like a follow-up question of: <a href="https://stackoverflow.com/questions/37138499/recommended-way-to-persistently-change-kube-env-variables">Recommended way to persistently change kube-env variables</a></p>

<p>I was playing around with the possibility to define a <em>k8s-user-startup-script</em> for GKE instances (I want to install additional software to each node).</p>

<p>Adding  <em>k8s-user-startup-script</em> to an Instance Group Template "Custom Metadata" works, but that is overwritten by <code>gcloud container clusters upgrade</code> which creates a new Instance Template without "inheriting" the additional <em>k8s-user-startup-script</em> Metadata from the current template.</p>

<p>I've also tried to add a <em>k8s-user-startup-script</em> to the project metadata (I thought that would be inherited by all instances of my project like described  <a href="https://cloud.google.com/compute/docs/metadata?hl=en_US#projectwide" rel="nofollow noreferrer">here</a>) but that is not taken into account.</p>

<p>What is the correct way to define a <em>k8s-user-startup-script</em> that persists cluster upgrades?
Or, more general, what is the desired way to customize the GKE nodes?</p>

<p>Google Container Engine doesn't support custom startup scripts for nodes. </p>

<p>As I mentioned in <a href="https://stackoverflow.com/questions/37138499/recommended-way-to-persistently-change-kube-env-variables">Recommended way to persistently change kube-env variables</a> you can use a DaemonSet to customize your nodes. A DaemonSet running in privileged mode can do pretty much anything that you could do with a startup script, with the caveat that it is done slightly later in the node bring-up lifecycle. Since a DaemonSet will run on all nodes in your cluster, it will be automatically applied to any new nodes that join (via cluster resize) and because it is a Kubernetes API object, it will be persisted across OS upgrades. </p>
