<p>I'm trying to deploy a model on GKE with tensorflow model serving using a GPU. I created a container with docker and it works great on a cloud VM. I'm trying to scale using GKE but the deployment exists with above error.</p>

<p>I created the GKE cluster with only 1 node, with a GPU (Tesla T4). I installed the drivers according to the <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#installing_drivers" rel="nofollow noreferrer">docs</a></p>

<p>It seems to be successful as much as I understand (a pod named <code>nvidia-driver-installer-tckv4</code> was added to the pods list in the node, and it's running without errors)</p>

<p>Next I created the deployment:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: reph-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: reph
    spec:
      containers:
      - name: reph-container
        image: gcr.io/&lt;project-id&gt;/reph_serving_gpu
        resources:
          limits:
            nvidia.com/gpu: 1
        ports:
        - containerPort: 8500
        args:
        - "--runtime=nvidia"
</code></pre>

<p>Then I ran kubectl create -f d1.yaml and the container exited with the above error in the logs.</p>

<p>I also tried to switch the os from cos to ubuntu and run an example from the <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#pods_gpus" rel="nofollow noreferrer">docs</a></p>

<p>I installed the drivers as above, this time for ubuntu. and applied this yaml taken from the GKE docs (only changed number of gpus to consume):</p>

<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: my-gpu-pod
spec:
  containers:
  - name: my-gpu-container
    image: nvidia/cuda:10.0-runtime-ubuntu18.04
    resources:
      limits:
       nvidia.com/gpu: 1
</code></pre>

<p>This time i'm getting CrashLoopBackOff without anything more in the logs.</p>

<p>Any idea wha'ts wrong? I'm a total newcomer to kubernetes and docker so I may be missing something trivial, but I really tried to stick to the GKE docs.</p>

<p>ok, I think the docs aren't clear enough on this, but it seems that what was missing was including <code>/usr/local/nvidia/lib64</code> in the <code>LD_LIBRARY_PATH</code> environment variable. 
The following yaml file runs successfully:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: reph-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: reph
    spec:
      containers:
      - name: reph-container
        env: 
        - name: LD_LIBRARY_PATH
          value: "$LD_LIBRARY_PATH:/usr/local/nvidia/lib64"
        image: gcr.io/&lt;project-id&gt;/reph_serving_gpu
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            nvidia.com/gpu: 1
        ports:
        - containerPort: 8500
        args:
        - "--runtime=nvidia"
</code></pre>

<p>Here's the relevant part in the GKE <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#cuda" rel="nofollow noreferrer">docs</a> </p>
