<p>I'm running a google cloud composer GKE cluster. I have a default node pool of 3 normal CPU nodes and one nodepool with a GPU node. The GPU nodepool has autoscaling activated.</p>
<p>I want to run a script inside a docker container on that GPU node.</p>
<p>For the GPU operating system I decided to go with cos_containerd instead of ubuntu.</p>
<p>I've followed <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/gpus" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/how-to/gpus</a> and ran this line:</p>
<pre><code>kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml
</code></pre>
<p>The GPU now shows up when I run &quot;kubectl describe&quot; on the GPU node, however my test scripts debug information tells me, that the GPU is not being used.</p>
<p>When I connect to the autoprovisioned GPU node via ssh, I can see, that I still need to run the</p>
<pre><code>cos extensions gpu install
</code></pre>
<p>in order to use the GPU.</p>
<p>I now want to make my cloud composer GKE cluster to run &quot;cos-extensions gpu install&quot; whenever a node is being created by the autoscaler feature.</p>
<p>I would like to apply something like this yaml:</p>
<pre><code>#cloud-config

runcmd:
  - cos-extensions install gpu
</code></pre>
<p>to my cloud composer GKE cluster.</p>
<p>Can i do that with kubectl apply ? Ideally I would like to only run that yaml code onto the GPU node. How can I achieve that?</p>
<p>I'm new to Kubernetes and I've already spent a lot of time on this without success. Any help would be much appreciated.</p>
<p>Best,
Phil</p>
<p><strong>UPDATE:</strong>
ok thx to Harsh I realized I have to go via Daemonset + ConfigMap like here:
<a href="https://github.com/GoogleCloudPlatform/solutions-gke-init-daemonsets-tutorial" rel="nofollow noreferrer">https://github.com/GoogleCloudPlatform/solutions-gke-init-daemonsets-tutorial</a></p>
<p>My GPU node has the label</p>
<pre><code>gpu-type=t4
</code></pre>
<p>so I've created and kubectl applied this ConfigMap:</p>
<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: phils-init-script
  labels:
    gpu-type: t4
data:
  entrypoint.sh: |
    #!/usr/bin/env bash

    ROOT_MOUNT_DIR=&quot;${ROOT_MOUNT_DIR:-/root}&quot;

    chroot &quot;${ROOT_MOUNT_DIR}&quot; cos-extensions gpu install
</code></pre>
<p>and here is my DaemonSet (I also kubectl applied this one):</p>
<pre><code>apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: phils-cos-extensions-gpu-installer
  labels:
    gpu-type: t4
spec:
  selector:
    matchLabels:
      gpu-type: t4
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: phils-cos-extensions-gpu-installer
        gpu-type: t4
    spec:
      volumes:
      - name: root-mount
        hostPath:
          path: /
      - name: phils-init-script
        configMap:
          name: phils-init-script
          defaultMode: 0744
      initContainers:
      - image: ubuntu:18.04
        name: phils-cos-extensions-gpu-installer
        command: [&quot;/scripts/entrypoint.sh&quot;]
        env:
        - name: ROOT_MOUNT_DIR
          value: /root
        securityContext:
          privileged: true
        volumeMounts:
        - name: root-mount
          mountPath: /root
        - name: phils-init-script
          mountPath: /scripts
      containers:
      - image: &quot;gcr.io/google-containers/pause:2.0&quot;
        name: pause
</code></pre>
<p>but nothing happens, i get the message &quot;Pods are pending&quot;.</p>
<p>During the run of the script I connect via ssh to the GPU node and can see that the ConfigMap shell code didn't get applied.</p>
<p>What am I missing here?</p>
<p>I'm desperately trying to make this work.</p>
<p>Best,
Phil</p>
<p>Thanks for all your help so far!</p>

<blockquote>
<p>Can i do that with kubectl apply ? Ideally I would like to only run
that yaml code onto the GPU node. How can I achieve that?</p>
</blockquote>
<p>Yes, You can run the Deamon set on each node which will run the command on Nodes.</p>
<p>As you are on GKE and Daemon set will also run the command or script on New nodes also which are getting scaled up also.</p>
<p>Daemon set is mainly for running applications or deployment on each available node in the cluster.</p>
<p>We can leverage this deamon set and run the command on each node that exist and is also upcoming.</p>
<p>Example <strong>YAML</strong> :</p>
<pre><code>apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-initializer
  labels:
    app: default-init
spec:
  selector:
    matchLabels:
      app: default-init
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: node-initializer
        app: default-init
    spec:
      volumes:
      - name: root-mount
        hostPath:
          path: /
      - name: entrypoint
        configMap:
          name: entrypoint
          defaultMode: 0744
      initContainers:
      - image: ubuntu:18.04
        name: node-initializer
        command: [&quot;/scripts/entrypoint.sh&quot;]
        env:
        - name: ROOT_MOUNT_DIR
          value: /root
        securityContext:
          privileged: true
        volumeMounts:
        - name: root-mount
          mountPath: /root
        - name: entrypoint
          mountPath: /scripts
      containers:
      - image: &quot;gcr.io/google-containers/pause:2.0&quot;
        name: pause
</code></pre>
<p>Github link for example : <a href="https://github.com/GoogleCloudPlatform/solutions-gke-init-daemonsets-tutorial" rel="nofollow noreferrer">https://github.com/GoogleCloudPlatform/solutions-gke-init-daemonsets-tutorial</a></p>
<p>Exact deployment step : <a href="https://cloud.google.com/solutions/automatically-bootstrapping-gke-nodes-with-daemonsets#deploying_the_daemonset" rel="nofollow noreferrer">https://cloud.google.com/solutions/automatically-bootstrapping-gke-nodes-with-daemonsets#deploying_the_daemonset</a></p>
<p>Full article : <a href="https://cloud.google.com/solutions/automatically-bootstrapping-gke-nodes-with-daemonsets" rel="nofollow noreferrer">https://cloud.google.com/solutions/automatically-bootstrapping-gke-nodes-with-daemonsets</a></p>
