<p>With the following kustomize:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    service: postfix
  name: postfix
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      service: postfix
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        service: postfix
    spec:
      containers:
      - name: postfix
        image: fk-postfix-placeholder
        command: [&quot;postfix&quot;]
        imagePullPolicy: Always
        env:
          - name: SMTP_DOMAIN
            valueFrom:
              secretKeyRef:
                name: my-environment-config
                key: SMTP_DOMAIN
          - name: SMTP_USER
            valueFrom:
              secretKeyRef:
                name: my-environment-config
                key: SMTP_USER
          - name: SMTP_PASSWORD
            valueFrom:
              secretKeyRef:
                name: my-environment-config
                key: SMTP_PASSWORD
        ports:
        - containerPort: 25
        resources:
          requests:
            cpu: 1m
            memory: 50Mi
        volumeMounts:
        - mountPath:  /var/lib/postfix
          name: my-storage
          subPath: postfix
        - mountPath: /dev/shm
          name: dshm

      restartPolicy: Always
      volumes:
      - name: my-storage
        persistentVolumeClaim:
          claimName: my-storage-claim-postfix
      - name: dshm
        emptyDir:
          medium: Memory
</code></pre>
<p>My pod does not come up:</p>
<pre><code>$k get pods

NAME                          READY   STATUS             RESTARTS   AGE
...
postfix-5555558c74-2kcv5      0/1     CrashLoopBackOff   16         18m
...
</code></pre>
<p>When I inspect it, I see the following in events</p>
<pre><code>$k describe pods/postfix-5555558c74-2kcv5
...
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  28m                   default-scheduler  Successfully assigned my-test/postfix-5555558c74-2kcv5 to gke-xxxxxxxxx-us-central1-default-pool-xxxxxx61-29fq
  Normal   Pulled     28m (x2 over 28m)     kubelet            Successfully pulled image &quot;registry.gitlab.com/octomy/postfix:1.0.0-test-development&quot;
  Normal   Pulling    27m (x3 over 28m)     kubelet            Pulling image &quot;registry.gitlab.com/octomy/postfix:1.0.0-test-development&quot;
  Normal   Created    27m (x3 over 28m)     kubelet            Created container postfix
  Normal   Started    27m (x3 over 28m)     kubelet            Started container postfix
  Warning  BackOff    3m8s (x125 over 28m)  kubelet            Back-off restarting failed container
</code></pre>
<p>But when I try to figure out what happened in logs, this is what I see:</p>
<pre><code>$k logs pods/postfix-5555558c74-2kcv5
</code></pre>
<p>Yes literally nothing is the content of the logs. When I run this in a local container using docker-compose the log is chock full of output.</p>
<p>So I want to know, what should be my next step in debuggin this?</p>

Mr. Developerdude
<p>The answer is obvious once you spot it. The important part is; <em>what does it mean when a normally log-happy container exits without a single line of log output?</em></p>
<p>The answer is, when a chatty container does not output any logs, <em>the container does not start at all</em>. This could have many causes, but in my case the reason was that I specified a custom <code>command</code> like so:</p>
<pre><code>command: [&quot;postfix&quot;]
</code></pre>
<p>There is no executable inside the container called &quot;postfix&quot; and so the the container simply terminated silently with exit code 1.</p>
<p>The reason why I had set a <code>command</code> in the first place was that I misunderstood the nomenclaturic differences between Docker and Kubernetes;</p>
<p>In <strong>Docker</strong> an <code>endpoint</code> is the executable to run and <code>command</code> is whatever arguments you pass to it. Both can be specified inside the image itself and both can be overridden at run time (for example on the commandline or in <code>docker-compose.yaml</code>).</p>
<p>In <strong>Kubernetes</strong> a <code>command</code> is the executable to run and <code>args</code> are the arguments you pass to it.</p>
<p>I wanted to override the Command specified inside the Docer container so I naively put <code>command: [&quot;postfix&quot;]</code> in my kubernetes yaml but of course in kubernetes this would override the executable rather than the arguments. The fix was to put <code>args: [&quot;postfix&quot;]</code> instead.</p>
<p>&lt;rant&gt;</p>
<p>IMHO I think this is an area of potential improvement on the part of kubernetes tools. There should at least be some kind of warning &quot;command specified does not seem to exist in container&quot; in the event log or appended to the empty log output.</p>
<p>&lt;/rant&gt;</p>
