<p>I am using Kubernetes to deploy a service that uses a local database. The service is deployed as a stateful set with 3 replicas. I have 3 different init containers but the 3rd container always fails with a crashLoopBackOff. The 3rd init container simply removes some directories on a mounted volume.  I've tried using multiple variations of removing the directory (if it exists) with the combination of bash logic or just simply <code>rm -rf</code>. The result is the same <code>crashLoopBackOff</code> without logs.</p>
<p><em>The Specific Init Container that is failing:</em></p>
<pre><code>- name: init-snapshot
        image: camlcasetezos/tezos:mainnet
        command: 
        - sh
        - -c
        # - exit 0
        - if [ -d &quot;/mnt/nd/node/data/store&quot; ]; then rm -Rf /mnt/nd/node/data/store; fi
        - if [ -d &quot;/mnt/nd/node/data/context&quot; ]; then rm -Rf /mnt/nd/node/data/context; fi
        volumeMounts:
        - name: node-data
          mountPath: /mnt/nd
        securityContext:
          runAsUser: 100
</code></pre>
<p><em>The entire StatefulSet:</em></p>
<pre><code>apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mainnet-full-node
  labels:
    app: mainnet-full
    component: mainnet-full-node
spec:
  serviceName: mainnet-full-rpc
  replicas: 3
  selector:
    matchLabels:
      app: mainnet-full
      component: mainnet-full-node
  template:
    metadata:
      labels:
        app: mainnet-full
        component: mainnet-full-node
    spec:
      initContainers:
      - name: init-perm
        # Fix the permissions of the storage volumes--chown to the right user.
        image: library/busybox
        command: 
        - sh
        - -c
        - chown -R 100 /mnt/*
        volumeMounts:
        - name: node-data
          mountPath: /mnt/nd
        - name: node-home
          mountPath: /mnt/nh
        securityContext:
          runAsUser: 0
      - name: init-identity
        # Generate a network identity if needed (use to repair the default, then disable)
        image: camlcasetezos/tezos:mainnet
        command: 
        - sh
        - -c
        - exit 0; rm /mnt/nd/node/data/identity.json 2&gt;&amp;1 &gt; /dev/null; /usr/local/bin/tezos-node identity generate 26 --data-dir=/mnt/nd/node/data
        volumeMounts:
        - name: node-data
          mountPath: /mnt/nd
        securityContext:
          runAsUser: 100
      - name: init-snapshot
        # Generate a network identity if needed (use to repair the default, then disable)
        image: camlcasetezos/tezos:mainnet
        command: 
        - sh
        - -c
        # - exit 0
        - if [ -d &quot;/mnt/nd/node/data/store&quot; ]; then rm -Rf /mnt/nd/node/data/store; fi
        - if [ -d &quot;/mnt/nd/node/data/context&quot; ]; then rm -Rf /mnt/nd/node/data/context; fi
        volumeMounts:
        - name: node-data
          mountPath: /mnt/nd
        securityContext:
          runAsUser: 100
      # We have to use host networking to get the correct address advertised?
      #hostNetwork: true
      containers:
      - name: mainnet-full-node
        image: camlcasetezos/tezos:mainnet
        args: [&quot;tezos-node&quot;, &quot;--history-mode&quot;, &quot;full&quot;]
        command: # Note the rpc address; block it from your firewall.
        - sh
        - -c
        - /usr/local/bin/tezos-node snapshot import /tmp/mainnet.full --data-dir=/var/run/tezos/node/data
        ports:
        - containerPort: 8732 # management
        - containerPort: 9732 # p2p service
        volumeMounts:
        - name: node-data
          mountPath: &quot;/var/run/tezos&quot;
        - name: node-home
          mountPath: &quot;/home/tezos&quot;
        - name: node-config
          mountPath: /home/tezos/.tezos-node
        - name: local-client-config
          mountPath: /home/tezos/.tezos-client
        securityContext:
          # emperically, this is the uid that gets chosen for the 'tezos'
          # user. Make it explicit.
          runAsUser: 100
      volumes:
      - name: node-data
        persistentVolumeClaim:
          claimName: node-data
      - name: node-config
        configMap:
          name: configs
          items:
          - key: node-config
            path: config
      - name: local-client-config
        configMap:
          name: configs
          items:
          - key: local-client-config
            path: config
  volumeClaimTemplates:
  - metadata:
      name: node-data
    spec:
      accessModes:
      - ReadWriteOnce
      volumeMode: Filesystem
      resources:
        requests:
          storage: 100Gi
      storageClassName: do-block-storage
  - metadata:
      name: node-home
    spec:
      accessModes:
      - ReadWriteOnce
      volumeMode: Filesystem
      resources:
        requests:
          storage: 1Gi
      storageClassName: do-block-storage

</code></pre>

Brice Aldrich
<p>Try using <code>kubectl logs -p podname</code> to get the previous logs.</p>
<p>Since it is in crash loop back off you’re seeing the pod logs before it crashes only.</p>
<p>If that doesn't work, then try <code>kubectl describe pod podname</code> and look at the events shown at the bottom. Usually if something is in CrashLoopBackoff there will at least be something in the events, even if the pod itself was never able to start.</p>
