<p>I am running MongoDB as a StatefulSet in Kubernetes.</p>
<p>I am trying to use startup/liveness probes, and I noticed some helm charts use the MongoDB &quot;ping&quot; command.</p>
<p>As <a href="https://docs.mongodb.com/manual/reference/command/ping/" rel="nofollow noreferrer">the documentation</a> says,</p>
<blockquote>
<p>The ping command is a no-op used to test whether a server is responding to commands. This command will return immediately even if the server is write-locked:</p>
</blockquote>
<p>What does it mean? When a server is starting or in the midst of initial sync, what will the command return?</p>

C C H
<p>I am not sure if the ping is a good idea. You don't care about the general state of the server; you care that it can receive connections.</p>
<p>Liveness probes have a timeout, so it's possible that in the future, when you're starting a new replica, the new pod in the stateful set will fail while waiting for the replication to end.</p>
<p>You should use the <a href="https://docs.mongodb.com/manual/reference/command/replSetGetStatus/#mongodb-dbcommand-dbcmd.replSetGetStatus" rel="nofollow noreferrer">rs.status()</a> and get the &quot;myState&quot; field.</p>
<p><em>myState</em> is an integer flag between 0-10. See <a href="https://docs.mongodb.com/manual/reference/replica-states/" rel="nofollow noreferrer">this</a> for all the possible statuses.</p>
<p>And if, for whatever reason the <em>rs.status()</em> command fails, that means that the ping would also fail.</p>
<p>However, a successful ping doesn't mean that the server is ready to receive connections and serve data, which is what you really care about.</p>
<h3>Probes</h3>
<p><strong>startup probe</strong>, myState equals to 1 or 2</p>
<p>This means that the startup probe will wait patiently until the server is ready, regardless if it's a primary or replica.</p>
<p><strong>Readiness probes</strong>, myState equals to 1 or 2</p>
<p>This means that, whenever a replica needs to rollback or is recovering or whatever reason that <em>mongod</em> decides that it's not ready to accept connections or serve data, this will let Kubernetes know that this pod is not ready, and will route requests to other pods in the sts.</p>
<p><strong>Livenes probe</strong>, myState is NOT equals to 6, 8 or 10</p>
<p>That means that, unless the server status is UNKOWN, DOWN or REMOVED, Kubernetes will assume that this server is alive.</p>
<p>So, let's test a scenario!</p>
<ul>
<li>sts started, first pod is on STARTUP, and myState = 0</li>
<li>startup probe waits</li>
<li>the first MongoDB node is ready, myState = 1</li>
<li>the startup probe finally passed. Now readiness and liveness probes start acting</li>
<li>a new replica triggered, second pod is on STARTUP, and myState = 0</li>
<li>a new replica successfully joins the set. myState = 5</li>
<li>a new replica is ready. myState = 2</li>
<li>the startup probe finally passed. Now readiness and liveness probes start acting</li>
<li><em>time for some action</em></li>
<li>a massive operation that altered hundreds of documents was rolled back on the primary</li>
<li>the second pod is now on ROLLBACK, myState = 9, and readiness probe failed. The second pod is now NOT READY</li>
<li>all connections are now sent to the PRIMARY</li>
<li>the second pod has finished the rollback</li>
<li>the second pod is now back as a SECONDARY, myState = 2, liveness probe succeeds and pod is back at the READY state</li>
<li>the MongoDB dba messed up and issued a command that removed the secondary from the replicationset, myState = 10</li>
<li>liveness probe fails, and Kubernetes kills the pod</li>
<li>sts wants two replicas and starts a second pod again
...</li>
</ul>
<p>All good :)</p>
