<p>I have an application running in Kubernetes as a <code>StatefulSet</code> that starts 2 pods. It has configured a liveness probe and a readiness probe.</p>
<p>The <code>liveness probe</code> call a simple <code>/health</code> endpoint that responds when the server is done loading</p>
<p>The <code>readiness probe</code>, wait for some start-up job to complete. The job can take several minutes in some cases, and only when it finish the api of the application is ready to start accepting requests.</p>
<p>Even when the api is not available my app also run side jobs that don't depend on it, and I expect them to be done while the startup is happening too.</p>
<p><strong>Is it possible to force Kubernetes deployment to complete and deploy 2 pods, even when the readiness probe is still not passing?</strong></p>
<p>From the docs I get that the only effect of a readiness probe not passing is that the current pod won't be included as available in the loadbalancer service (which is actually the only effect that I want).</p>
<blockquote>
<p>If the readiness probe fails, the endpoints controller removes the
Pod's IP address from the endpoints of all Services that match the
Pod.</p>
</blockquote>
<p>However I am also seeing that the deployment never finishes, since pod 1 readiness probe is not passing and pod 2 is never created.</p>
<pre><code>kubectl rollout restart statefulset/pod
kubectl get pods 
NAME                            READY   STATUS    RESTARTS   AGE
pod-0                           1/2     Running   0          28m
</code></pre>
<p>If the readiness probe failure, always prevent the deployment, <strong>Is there other way to selectively expose only ready pods in the loadbalancer, while not marking them as Unready during the deployment?</strong></p>
<p>Thanks in advance!</p>

jesantana
<h2>StatefulSet deployment</h2>
<blockquote>
<p>Is it possible to force kubernetes deployment to complete and deploy 2
pods, even when the readiness probe is still not passing?</p>
</blockquote>
<p>Assuming it's meant <code>statefulSet</code> instead of <code>deployment</code> as object, the answer is no, it's not possible by design, most important is second point:</p>
<ul>
<li>For a StatefulSet with N replicas, when Pods are being deployed, they are created sequentially, in order from {0..N-1}.</li>
<li>Before a scaling operation is applied to a Pod, all of its predecessors must be Running and Ready.</li>
<li>When Pods are being deleted, they are terminated in reverse order, from {N-1..0}.</li>
</ul>
<blockquote>
<p>When the nginx example above is created, three Pods will be deployed
in the order web-0, web-1, web-2. web-1 will not be deployed before
web-0 is Running and Ready, and web-2 will not be deployed until web-1
is Running and Ready</p>
</blockquote>
<p><a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#deployment-and-scaling-guarantees" rel="nofollow noreferrer">StatefulSets - Deployment and scaling guaranties</a></p>
<h2>Readyness probe, endpoints and potential workaround</h2>
<blockquote>
<p>If the readiness probe failure, always prevent the deployment, Is
there other way to selectively expose only ready pods in the load
balancer, while not marking them as Unready during the deployment?</p>
</blockquote>
<p>This is by design, pods are added to service endpoints once they are in <code>ready</code> state.</p>
<p>Some kind of potential workaround can be used, at least in simple example it does work, however you should try and evaluate if this approach will suit your case, this is fine to use as initial deployment.</p>
<p><code>statefulSet</code> can be started without <code>readyness</code> probe included, this way <code>statefulSet</code> will start pods one by one when previous is <code>run and ready</code>, <code>liveness</code> may need to set up <code>initialDelaySeconds</code> so kubernetes won't restart the pod thinking it's unhealthy. Once <code>statefulSet</code> is fully run and ready, you can add <code>readyness</code> probe to the <code>statefulSet</code>.</p>
<p>When <code>readyness</code> probe is added, kubernetes will restart all pods again starting from the last one and your application will need to start again.</p>
<p>Idea is to start all pods and they will be able to serve requests +- at the same time, while with <code>readyness</code> probe applied, only one pod will start in 5 minutes for instance, next pod will take 5 minutes more and so on.</p>
<h2>Example</h2>
<p>Simple example to see what's going on based on <code>nginx</code> webserver and <code>sleep 30</code> command which makes kubernetes think when <code>readyness</code> probe is setup that pod is <code>not ready</code>.</p>
<ol>
<li>Apply <code>headless service</code></li>
<li>Comment <code>readyness</code> probe in <code>statefulSet</code> and apply manifest</li>
<li>Observe that all pods are created right after previous pod is <code>running and ready</code></li>
<li>Uncomment <code>readyness</code> probe and apply manifest</li>
<li>Kubernetes will recreate all pods starting from the last one waiting this time <code>readyness</code> probe to complete and flag a pod as <code>running and ready</code>.</li>
</ol>
<p>Very convenient to use this command to watch for progress:</p>
<pre><code>watch -n1 kubectl get pods -o wide
</code></pre>
<p><code>nginx-headless-svc.yaml</code>:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
</code></pre>
<p><code>nginx-statefulset.yaml</code>:</p>
<pre><code>apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  serviceName: &quot;nginx&quot;
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          name: web
        command: [&quot;/bin/bash&quot;, &quot;-c&quot;]
        args: [&quot;sleep 30 ; echo sleep completed ;  nginx -g \&quot;daemon off;\&quot;&quot;]
        readinessProbe:
          tcpSocket:
            port: 80
          initialDelaySeconds: 1
          periodSeconds: 5
</code></pre>
<h2>Update</h2>
<p>Thanks to @jesantana for this much easier solution.</p>
<p>If all pods have to be scheduled at once and it's not necessary to wait for pods readyness, <code>.spec.podManagementPolicy</code> can be set to <code>Parallel</code>. <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-management-policies" rel="nofollow noreferrer">Pod Management Policies</a></p>
<h2>Useful links:</h2>
<ul>
<li><a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/" rel="nofollow noreferrer">Kubernetes statefulsets</a></li>
<li><a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/" rel="nofollow noreferrer">kubernetes liveness, readyness and startup probes</a></li>
</ul>
