<p>While learning Kubernetes going by the book Kubernetes for developer, I am stuck at this point now.</p>
<p>I am trying to start Rabbitmq pod but but after lot of troubleshooting I have managed to get to this point but do not get clue where do I fix to get rid of the permission denied error.</p>
<pre><code># kubectl get pods
NAME                        READY   STATUS             RESTARTS   AGE
rabbitmq-56c67d8d7d-s8vp5   0/1     CrashLoopBackOff   5          5m40s
</code></pre>
<p>if I look at the logs of this contianer thats where I found:</p>
<pre><code># kubectl logs  rabbitmq-56c67d8d7d-s8vp5
 21:22:58.49 
 21:22:58.50 Welcome to the Bitnami rabbitmq container
 21:22:58.51 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-rabbitmq
 21:22:58.51 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-rabbitmq/issues
 21:22:58.52 Send us your feedback at containers@bitnami.com
 21:22:58.52 
 21:22:58.52 INFO  ==&gt; ** Starting RabbitMQ setup **
 21:22:58.54 INFO  ==&gt; Validating settings in RABBITMQ_* env vars..
 21:22:58.56 INFO  ==&gt; Initializing RabbitMQ...
 21:22:58.57 INFO  ==&gt; Generating random cookie
mkdir: cannot create directory ‘/bitnami/rabbitmq’: Permission denied
</code></pre>
<p>Here is my <strong>rabbitmq-deployment.yml</strong></p>
<pre><code>---
# EXPORT SERVICE INTERFACE
kind: Service
apiVersion: v1
metadata:
    name: message-queue
    labels:
      app: rabbitmq
      role: master
      tier: queue
spec:
  ports:
  - port: 5672
    targetPort: 5672
  selector:
      app: rabbitmq
      role: master
      tier: queue
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rabbitmq-pv-claim
  labels:
    app: rabbitmq
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rabbitmq
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rabbitmq
      role: master
      tier: queue
  template:
    metadata:
      labels:
         app: rabbitmq
         role: master
         tier: queue
    spec:
      nodeSelector:
        boardType: x86vm
      containers:
      - name: rabbitmq
        image: bitnami/rabbitmq:3.7
        envFrom:
        - configMapRef:
            name: bitnami-rabbitmq-config
        ports:
        - name: queue
          containerPort: 5672
        - name: queue-mgmt
          containerPort: 15672
        livenessProbe:
          exec:
            command:
            - rabbitmqctl
            - status
          initialDelaySeconds: 120
          timeoutSeconds: 5
          failureThreshold: 6
        readinessProbe:
          exec:
            command:
            - rabbitmqctl
            - status
          initialDelaySeconds: 10
          timeoutSeconds: 3
          periodSeconds: 5
        volumeMounts:
        - name: rabbitmq-storage
          mountPath: /bitnami
      volumes:
      - name: rabbitmq-storage
        persistentVolumeClaim:
          claimName: rabbitmq-pv-claim
</code></pre>
<p>This is the <strong>rabbitmq-storage-class.yml</strong></p>
<pre><code>kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: rabbitmq-storage-class
  labels:
    app: rabbitmq
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
</code></pre>
<p>and <strong>persistant-volume.yml</strong></p>
<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: rabbitmq-pv-claim
  labels:
    app: rabbitmq
spec:
  storageClassName: manual
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  hostPath:
    path: /bitnami
</code></pre>
<p>Logs:</p>
<pre><code># kubectl describe pods rabbitmq-5f7f787479-fpg6g
Name:         rabbitmq-5f7f787479-fpg6g
Namespace:    default
Priority:     0
Node:         kube-worker-vm2/192.168.1.36
Start Time:   Mon, 03 May 2021 12:29:17 +0100
Labels:       app=rabbitmq
              pod-template-hash=5f7f787479
              role=master
              tier=queue
Annotations:  cni.projectcalico.org/podIP: 192.168.222.4/32
              cni.projectcalico.org/podIPs: 192.168.222.4/32
Status:       Running
IP:           192.168.222.4
IPs:
  IP:           192.168.222.4
Controlled By:  ReplicaSet/rabbitmq-5f7f787479
Containers:
  rabbitmq:
    Container ID:   docker://bbdbb9c5d4b6737519d3dcf4bdda242a7fe904f2336334afe686e9b204fd6d5c
    Image:          bitnami/rabbitmq:3.7
    Image ID:       docker-pullable://bitnami/rabbitmq@sha256:8b6057997b74ebc81e934dd6c94e9da745635faa2d79b382cfda27b9176e0e6d
    Ports:          5672/TCP, 15672/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 03 May 2021 12:30:48 +0100
      Finished:     Mon, 03 May 2021 12:30:48 +0100
    Ready:          False
    Restart Count:  4
    Liveness:       exec [rabbitmqctl status] delay=120s timeout=5s period=10s #success=1 #failure=6
    Readiness:      exec [rabbitmqctl status] delay=10s timeout=3s period=5s #success=1 #failure=3
    Environment Variables from:
      bitnami-rabbitmq-config  ConfigMap  Optional: false
    Environment:               &lt;none&gt;
    Mounts:
      /bitnami from rabbitmq-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-4qmxr (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  rabbitmq-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  rabbitmq-pv-claim
    ReadOnly:   false
  default-token-4qmxr:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-4qmxr
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  boardType=x86vm
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  2m20s                 default-scheduler  Successfully assigned default/rabbitmq-5f7f787479-fpg6g to kube-worker-vm2
  Normal   Created    96s (x4 over 2m18s)   kubelet            Created container rabbitmq
  Normal   Started    95s (x4 over 2m17s)   kubelet            Started container rabbitmq
  Warning 

 BackOff    65s (x12 over 2m16s)  kubelet            Back-off restarting failed container
  Normal   Pulled     50s (x5 over 2m18s)   kubelet            Container image &quot;bitnami/rabbitmq:3.7&quot; already present on machine
</code></pre>

<p>When creating an image, the image creator often chooses to use a user other than root to run the process. This is the case for your image, and the user does not have write permissions on the <code>/bitnami</code> directory. You can verify this by commenting out the volume.</p>
<p>To fix the issue, you need to set a security contect for your pod: <a href="https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod</a></p>
<p>Not sure about the exact syntax, but something like this should do the trick:</p>
<pre><code>spec:
  securityContext:
    fsGroup: 1001 # the userid that is used in the image
  nodeSelector:
    boardType: x86vm
  containers:
  - name: rabbitmq
    image: bitnami/rabbitmq:3.7
    envFrom:
    - configMapRef:
        name: bitnami-rabbitmq-config
</code></pre>
<p>This makes the directory writeable by the user in the image.</p>
<p>Another thing: A deployment is for stateless services by design. If you have state to keep, always use a statefulset. It's very similiar to a deployment from a configuration point of view, but Kubernetes treats it very differently. See <a href="https://www.youtube.com/watch?v=Vrxr-7rjkvM" rel="nofollow noreferrer">https://www.youtube.com/watch?v=Vrxr-7rjkvM</a> for good explanation.</p>
