<p>I have setup a mongodb workload in Rancher (2.5.8)</p>
<p>I have setup a volume:
<a href="https://i.stack.imgur.com/PgCm4.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/PgCm4.png" alt="volume" /></a></p>
<p>The workload start fine if I have the containers set to scale to 1.  So 1 container will start and all is fine.</p>
<p>However if I set the workload to have 2 or more containers, one container will start fine, but then the others fail to start.</p>
<p>Here is what my workload looks like if I set it to scale to 2.  one container started and running fine, but the second (and third if I have its scale to 3) are failing.
<a href="https://i.stack.imgur.com/VbwjH.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VbwjH.png" alt="enter image description here" /></a></p>
<p>If I remove the volume, then 2+ containers will all start up fine, but then data is only being stored within each container (and gets lost whenever I redeploy).</p>
<p>But if I have the volume set, then the data does store in the volume (host), but then can only start one container.</p>
<p>Thank you in advance for any suggestions</p>
<p>Jason</p>

Jason
<p>Posting this community wiki answer to set a baseline and to hopefully show one possible reason that the <code>mongodb</code> is failing.</p>
<p>Feel free to edit/expand.</p>
<hr />
<p>As there is a lot of information missing from this question like how it was created, how the <code>mongodb</code> was provisioned and there is also lack of logs from the container, the actual issue could be hard to pinpoint.</p>
<p>Assuming that the <code>Deployment</code> was created with a following manifest:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongo
spec:
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  replicas: 1 # THEN SCALE TO 3
  selector:
    matchLabels:
      app: mongo  
  template:
    metadata:
      labels:
        app: mongo
    spec:
      containers:
      - name: mongo
        image: mongo
        imagePullPolicy: &quot;Always&quot;
        ports:
        - containerPort: 27017
        volumeMounts:
        - mountPath: /data/db
          name: mongodb
      volumes:
      - name: mongodb
        persistentVolumeClaim:
          claimName: mongo-pvc
</code></pre>
<p>The part of the setup where the <code>Volume</code> is referenced could be different (for example <code>hostPath</code> can be used) but the premise of it was:</p>
<ul>
<li>If the <code>Pods</code> are physically referencing the same <code>data/db/mongod</code> it will go into <code>CrashLoopBackOff</code> state.</li>
</ul>
<p>Following on this topic:</p>
<ul>
<li><code>$ kubectl get pods</code></li>
</ul>
<pre class="lang-sh prettyprint-override"><code>NAME                     READY   STATUS             RESTARTS   AGE
mongo-5d849bfd8f-8s26t   1/1     Running            0          45m
mongo-5d849bfd8f-l6dzb   0/1     CrashLoopBackOff   13         44m
mongo-5d849bfd8f-wgh6m   0/1     CrashLoopBackOff   13         44m
</code></pre>
<ul>
<li><code>$ kubectl logs mongo-5d849bfd8f-l6dzb</code></li>
</ul>
<pre class="lang-sh prettyprint-override"><code>&lt;-- REDACTED --&gt; 
{&quot;t&quot;:{&quot;$date&quot;:&quot;2021-06-05T12:43:58.025+00:00&quot;},&quot;s&quot;:&quot;E&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:20557,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;DBException in initAndListen, terminating&quot;,&quot;attr&quot;:{&quot;error&quot;:&quot;DBPathInUse: Unable to lock the lock file: /data/db/mongod.lock (Resource temporarily unavailable). Another mongod instance is already running on the /data/db directory&quot;}}
&lt;-- REDACTED --&gt; 
</code></pre>
<hr />
<p>Citing the O`Reilly site on the <code>mongodb</code> production setup:</p>
<blockquote>
<p>Specify an alternate directory to use as the data directory; the default is <code>/data/db/</code> (or, on Windows, <code>\data\db\</code> on the MongoDB binary’s volume). Each mongod process on a machine needs its own data directory, so if you are running three instances of <code>mongod</code> on one machine, you’ll need three separate data directories. When <code>mongod</code> starts up, it creates a <code>mongod.lock</code> file in its data directory, which prevents any other <code>mongod</code> process from using that directory. If you attempt to start another MongoDB server using the same data directory, it will give an error:</p>
<blockquote>
<pre><code>exception in initAndListen: DBPathInUse: Unable to lock the
     lock file: \ data/db/mongod.lock (Resource temporarily unavailable).
     Another mongod instance is already running on the 
     data/db directory,
     \ terminating`
</code></pre>
</blockquote>
<p>-- <em><a href="https://www.oreilly.com/library/view/mongodb-the-definitive/9781491954454/ch21.html" rel="nofollow noreferrer">Oreilly.com: Library: View: Mongodb the definitive: Chapter 21</a></em></p>
</blockquote>
<hr />
<p>As an alternative approach you can other means to provision <code>mongodb</code> like for example:</p>
<ul>
<li><em><a href="https://docs.mongodb.com/kubernetes-operator/master/tutorial/deploy-replica-set/" rel="nofollow noreferrer">Docs.mongodb.com: Kubernetes operator: Master: Tutorial: Deploy replica set</a></em> (I would check the configuration of <code>StorageClasses</code> here)</li>
<li><em><a href="https://bitnami.com/stack/mongodb/helm" rel="nofollow noreferrer">Bitname.com: Stack: Mongodb: Helm</a></em></li>
</ul>
