<p>I am attempting to get a (currently single-node) RabbitMQ cluster created inside of a local Minikube instance. However, there seems to be a permission issue when attempting to create the RMQ cluster on Minikube with two nodes.</p>
<p>Prerequisites:</p>
<ol>
<li>Have Minikube, kubectl, and krew installed.</li>
</ol>
<p>Steps to reproduce:</p>
<ol>
<li>Start Minikube <code>minikube start --memory 8192 --cpus 4 --nodes 2</code></li>
</ol>
<pre><code>😄  minikube v1.27.1 on Debian bookworm/sid
✨  Automatically selected the docker driver
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=4, Memory=8192MB) ...
🐳  Preparing Kubernetes v1.25.2 on Docker 20.10.18 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass

👍  Starting worker node minikube-m02 in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=4, Memory=8192MB) ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.49.2
🐳  Preparing Kubernetes v1.25.2 on Docker 20.10.18 ...
    ▪ env NO_PROXY=192.168.49.2
🔎  Verifying Kubernetes components...
🏄  Done! kubectl is now configured to use &quot;minikube&quot; cluster and &quot;default&quot; namespace by default
</code></pre>
<ol start="2">
<li>Install the RabbitMQ plugin to kubectl <code>kubectl krew install rabbitmq</code></li>
<li>Install the RabbitMQ cluster operator into Minikube <code>kubectl rabbitmq install-cluster-operator</code></li>
<li>Create a default, single-node cluster with the operator <code>kubectl rabbitmq create default</code></li>
</ol>
<p>This results in the Persistent Volume (PV), Persistent Volume Claim (PVC), Stateful Set, and Service being created. After waiting a little while for the PVC to be attached to the PV, the Pod gets created. However, I end up with the following console errors with the Pod and the Pod starts a crash loop:</p>
<pre><code>2023-01-19 16:31:42.993395+00:00 [warning] &lt;0.130.0&gt; Failed to write PID file &quot;/var/lib/rabbitmq/mnesia/rabbit@default-server-0.default-nodes.default.pid&quot;: permission denied
2023-01-19 16:31:43.520923+00:00 [info] &lt;0.221.0&gt; Feature flags: list of feature flags found:
2023-01-19 16:31:43.520982+00:00 [info] &lt;0.221.0&gt; Feature flags:   [ ] classic_mirrored_queue_version
2023-01-19 16:31:43.521013+00:00 [info] &lt;0.221.0&gt; Feature flags:   [ ] implicit_default_bindings
2023-01-19 16:31:43.521060+00:00 [info] &lt;0.221.0&gt; Feature flags:   [ ] maintenance_mode_status
2023-01-19 16:31:43.521087+00:00 [info] &lt;0.221.0&gt; Feature flags:   [ ] quorum_queue
2023-01-19 16:31:43.521118+00:00 [info] &lt;0.221.0&gt; Feature flags:   [ ] stream_queue
2023-01-19 16:31:43.521147+00:00 [info] &lt;0.221.0&gt; Feature flags:   [ ] user_limits
2023-01-19 16:31:43.521186+00:00 [info] &lt;0.221.0&gt; Feature flags:   [ ] virtual_host_metadata
2023-01-19 16:31:43.521204+00:00 [info] &lt;0.221.0&gt; Feature flags: feature flag states written to disk: yes
2023-01-19 16:31:43.688848+00:00 [notice] &lt;0.44.0&gt; Application syslog exited with reason: stopped
2023-01-19 16:31:43.689010+00:00 [notice] &lt;0.221.0&gt; Logging: switching to configured handler(s); following messages may not be visible in this log output
2023-01-19 16:31:43.697297+00:00 [notice] &lt;0.221.0&gt; Logging: configured log handlers are now ACTIVE
2023-01-19 16:31:43.715974+00:00 [error] &lt;0.221.0&gt; 
2023-01-19 16:31:43.715974+00:00 [error] &lt;0.221.0&gt; BOOT FAILED
2023-01-19 16:31:43.715974+00:00 [error] &lt;0.221.0&gt; ===========
2023-01-19 16:31:43.715974+00:00 [error] &lt;0.221.0&gt; Error during startup: {error,
2023-01-19 16:31:43.715974+00:00 [error] &lt;0.221.0&gt;                           {cannot_create_mnesia_dir,
2023-01-19 16:31:43.715974+00:00 [error] &lt;0.221.0&gt;                               &quot;/var/lib/rabbitmq/mnesia/rabbit@default-server-0.default-nodes.default/&quot;,
2023-01-19 16:31:43.715974+00:00 [error] &lt;0.221.0&gt;                               eacces}}
2023-01-19 16:31:43.715974+00:00 [error] &lt;0.221.0&gt; 
BOOT FAILED
===========
Error during startup: {error,
                          {cannot_create_mnesia_dir,
                              &quot;/var/lib/rabbitmq/mnesia/rabbit@default-server-0.default-nodes.default/&quot;,
                              eacces}}
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;   crasher:
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;     initial call: application_master:init/4
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;     pid: &lt;0.220.0&gt;
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;     registered_name: []
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;     exception exit: {{cannot_create_mnesia_dir,
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;                          &quot;/var/lib/rabbitmq/mnesia/rabbit@default-server-0.default-nodes.default/&quot;,
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;                          eacces},
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;                      {rabbit,start,[normal,[]]}}
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;       in function  application_master:init/4 (application_master.erl, line 142)
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;     ancestors: [&lt;0.219.0&gt;]
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;     message_queue_len: 1
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;     messages: [{'EXIT',&lt;0.221.0&gt;,normal}]
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;     links: [&lt;0.219.0&gt;,&lt;0.44.0&gt;]
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;     dictionary: []
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;     trap_exit: true
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;     status: running
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;     heap_size: 987
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;     stack_size: 29
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;     reductions: 158
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt;   neighbours:
2023-01-19 16:31:44.716751+00:00 [error] &lt;0.220.0&gt; 
2023-01-19 16:31:44.720886+00:00 [notice] &lt;0.44.0&gt; Application rabbit exited with reason: {{cannot_create_mnesia_dir,&quot;/var/lib/rabbitmq/mnesia/rabbit@default-server-0.default-nodes.default/&quot;,eacces},{rabbit,start,[normal,[]]}}
{&quot;Kernel pid terminated&quot;,application_controller,&quot;{application_start_failure,rabbit,{{cannot_create_mnesia_dir,\&quot;/var/lib/rabbitmq/mnesia/rabbit@default-server-0.default-nodes.default/\&quot;,eacces},{rabbit,start,[normal,[]]}}}&quot;} 
Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{{cannot_create_mnesia_dir,&quot;/var/lib/rabbitmq/mnesia/rabbit@default-server-0.default-nodes.default/&quot;,eacces},{rabbit,start,[normal,[]]}}}) 
 
Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done 
</code></pre>
<hr />
<p><strong>However</strong>, if I delete my Minikube (<code>minikube stop &amp;&amp; minikube delete</code>) and re-create it with a single node (<code>minikube start --memory 8192 --cpus 4 --nodes 1</code>) and follow the aforementioned steps to create the default RabbitMQ cluster, then there are no problems. I don't understand how adding a second node to Minikube could cause this issue.</p>
<p>I feel like I'm just missing something obvious, but not sure what it is.</p>
<p>Any sort of suggestions or feedback would be greatly appreciated. Please let me know if there are more details I should provide. Thank you in advance!</p>

PhasedAndConfused
<p>In typical &quot;me&quot; fashion, I finally found the correct phrase to search for and get results. It turns out that the storage provisioner bundled with minikube doesn't really work with 2+ nodes. Replacing it with another one (kubevirt) as explained by this <a href="https://github.com/kubernetes/minikube/issues/12360#issuecomment-1123794143" rel="nofollow noreferrer">GitHub issue comment</a> allows the Pod to spin up correctly.</p>
<p>To add more context, in case that link goes bad someday:
I created a <code>kubevirt-hostpath-provisioner.yaml</code> file (contents given below) and then replaced the storage provisioner in minikube:</p>
<pre class="lang-bash prettyprint-override"><code>minikube addons disable storage-provisioner
kubectl delete storageclass standard
kubectl apply -f kubevirt-hostpath-provisioner.yaml
</code></pre>
<hr />
<pre class="lang-yaml prettyprint-override"><code># kubevirt-hostpath-provisioner.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: &quot;true&quot;
provisioner: kubevirt.io/hostpath-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubevirt-hostpath-provisioner
subjects:
  - kind: ServiceAccount
    name: kubevirt-hostpath-provisioner-admin
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: kubevirt-hostpath-provisioner
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubevirt-hostpath-provisioner
rules:
  - apiGroups: [&quot;&quot;]
    resources: [&quot;nodes&quot;]
    verbs: [&quot;get&quot;]
  - apiGroups: [&quot;&quot;]
    resources: [&quot;persistentvolumes&quot;]
    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;create&quot;, &quot;delete&quot;]
  - apiGroups: [&quot;&quot;]
    resources: [&quot;persistentvolumeclaims&quot;]
    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;update&quot;]

  - apiGroups: [&quot;storage.k8s.io&quot;]
    resources: [&quot;storageclasses&quot;]
    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]

  - apiGroups: [&quot;&quot;]
    resources: [&quot;events&quot;]
    verbs: [&quot;list&quot;, &quot;watch&quot;, &quot;create&quot;, &quot;update&quot;, &quot;patch&quot;]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubevirt-hostpath-provisioner-admin
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kubevirt-hostpath-provisioner
  labels:
    k8s-app: kubevirt-hostpath-provisioner
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: kubevirt-hostpath-provisioner
  template:
    metadata:
      labels:
        k8s-app: kubevirt-hostpath-provisioner
    spec:
      serviceAccountName: kubevirt-hostpath-provisioner-admin
      containers:
        - name: kubevirt-hostpath-provisioner
          image: quay.io/kubevirt/hostpath-provisioner
          imagePullPolicy: Always
          env:
            - name: USE_NAMING_PREFIX
              value: &quot;false&quot; # change to true, to have the name of the pvc be part of the directory
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: PV_DIR
              value: /tmp/hostpath-provisioner
          volumeMounts:
            - name: pv-volume # root dir where your bind mounts will be on the node
              mountPath: /tmp/hostpath-provisioner/
              #nodeSelector:
              #- name: xxxxxx
      volumes:
        - name: pv-volume
          hostPath:
            path: /tmp/hostpath-provisioner/

</code></pre>
