<p><strong>Description</strong>
Trying to deploy the triton docker image as container on kubernetes cluster</p>
<p><strong>Triton Information</strong>
What version of Triton are you using? -&gt; 22.10</p>
<p><strong>Are you using the Triton container or did you build it yourself?</strong>
I used the server repo with following command:</p>
<pre><code>python3 compose.py --backend onnxruntime --backend python --backend tensorflow2 --repoagent checksum --container-version 22.10
</code></pre>
<p>then again created new triton image with following dockerfile:</p>
<pre><code>FROM tritonserver:latest
RUN apt install python3-pip -y
RUN pip install tensorflow==2.7.0
RUN pip install transformers==2.11.0
RUN pip install tritonclient
RUN pip install tritonclient[all]
</code></pre>
<p>and dockerfile is being with following command:</p>
<pre><code>docker build -t customtritonimage -f ./DockerFiles/DockerFile  .
</code></pre>
<p><strong>To Reproduce</strong>
directory structure:
parent directory -&gt; tritonnludeployment
files in it -&gt; DockerFiles (folder containing docker files), k8_trial.yaml, model_repo_triton (all the models here in triton-supported directory shape and has required files)</p>
<p>I am using this 'k8_trial.yaml' file for starting kubectl deployment</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
    name: flower
    labels:
      app: flower
spec:
    replicas: 3
    selector:
      matchLabels:
        app: flower
    template:
      metadata:
        labels:
          app: flower
      spec:
        volumes:
        - name: models
          hostPath:
            # server: 216.48.183.17
            path: /root/Documents/tritonnludeployment
            # readOnly: false
            type: Directory
        containers:
          - name: flower
            ports:
            - containerPort: 8000
              name: http-triton
            - containerPort: 8001
              name: grpc-triton
            - containerPort: 8002
              name: metrics-triton
            image: &quot;customtritonimage:latest&quot;
            imagePullPolicy: Never
            volumeMounts:
            - mountPath: /root/Documents/tritonnludeployment
              name: models
            command: [&quot;/bin/sh&quot;, &quot;-c&quot;]
            args: [&quot;cd /models /opt/tritonserver/bin/tritonserver --model-repository=/models/model_repo_triton --allow-gpu-metrics=false --strict-model-config=false&quot;]
            # resources:
            #   requests:
            #     memory: &quot;500Mi&quot;
            #     cpu: &quot;500Mi&quot;
            #   limits:
            #     memory: &quot;900Mi&quot;
            #     cpu: &quot;900Mi&quot;
            #     nvidia.com/gpu: 1
</code></pre>
<p>Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).</p>
<p><strong>Expected behavior</strong>
kubectl deployment should start, with triton container as one of the pods</p>
<p>Which step i am doing wrong!</p>

<p>And what is the error message you are getting? Some of the issues I noticed:</p>
<ul>
<li>use the expected file name know to <code>docker</code>, i.e. <code>Dockerfile</code> not <code>DockerFile</code></li>
<li>make sure base image exists (<code>tritonserver:latest</code> does not, you probably want one of <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver/tags" rel="nofollow noreferrer">these</a>)</li>
<li>first update the sources (<code>RUN apt install</code> ... -&gt; <code>RUN apt update &amp;&amp; apt install </code>...)</li>
<li>reduce layers number by installing multiple python packages at once</li>
<li><code>tritonclient[all]</code> already includes <code>tritonclient</code></li>
<li>don't run containers as <code>root</code> (<code>tritonserver</code> does not require it anyway)</li>
<li>make sure you pull the image first time (<code>imagePullPolicy</code>: <code>Never</code> -&gt; <code>IfNotPresent</code>)</li>
<li>remove multiple and unnecessary commands from <code>args</code> (such as <code>cd /models</code>)</li>
<li><code>tritonserver</code> can import all subfolders, so <code>--model-repository=/models</code> is probably better</li>
</ul>
