<p>I have several pods where the rollout (initial and later updates) needs to performed one by one. (Actually only the first needs to be ready before the remaining can start or be upgraded)
I used stateful set for that, as it makes sure only one at a time is updated or created, but we are usung telepresence for development and it doesn’t support replacing stateful sets. So I thought I could use deployment instead of stateful set with rolling update strategy and limit the number of maxunavailable or maxsurge or whatever to “throttle” the deployment.
But for the initial deploymemt that doesn’t work, as K8s creates the desired 2 at once instead one one by one.
Is there a way to achieve that with a deploment or do I need to use a stateful set? (alternatively: is there a trick to use telepresence with stateful sets)</p>
<p>Clarification based on questions in comments:</p>
<ul>
<li>The problematic software here is flyway in combination with mariadb in cluster mode. Then the table locking doesn't work and simultaniously starting pods can try to perform schema and data updates at the same time</li>
<li>init containers don't help, as they start at the same time for multiple instances of the pod and just make sure that the main container of each instance is started after the init container</li>
<li>The problem is only on first initalization, because afterwards I can configure the rolling update strategy to only update one container at a time. In case of a scaling out I'd have to do it in increments of 1, but that would be a manual process anyway.</li>
<li>I could make sure that the deployment descriptor for new deployments uses scale 1 and updates to scale 2 afterwards, but that leads to a very complicated automatic deployment process with variable scales dependig on the state and the build chain would need to check if a deployment is present to decide if it's an update or a first deployment. Which would be error prone and overly complex</li>
</ul>

Patrick Cornelissen
<p>In my opinion, there are two possible solutions, but both require extra effort.<br />
I will describe both solutions and you can choose which one suits you best.</p>
<hr />
<h3>Solution one:  Deploying application with a script</h3>
<p>You can deploy your application with the script below:</p>
<pre><code>$ cat deploy.sh
#!/bin/bash

# Usage: deploy.sh DEPLOYMENT_FILENAME NAMESPACE NUMBER_OF_REPLICAS

deploymentFileName=$1   # Deployment manifest file name
namespace=$2            # Namespace where app should be deployed
replicas=$3             # Numbers of replicas that should be deployed

# First deploy ONLY one replica - sed command changes actual number of replicas to 1 but the original manifest file remains intact
cat ${deploymentFileName} | sed -E 's/replicas: [0-9]+$/replicas: 1/' | kubectl apply -n $namespace -f -

# The &quot;until&quot; loop waits until the first replica is ready
# Check deployment rollout status every 10 seconds (max 10 minutes) until complete.
attempts=0
rollout_cmd=&quot;kubectl rollout status -f ${deploymentFileName} -n $namespace&quot;
until $rollout_cmd || [ $attempts -eq 60 ]; do
    $rollout_cmd
    attempts=$((attempts + 1))
    sleep 10
done

if [ $attempts -eq 60 ]; then
    echo &quot;ERROR: Timeout&quot;
    exit 1
fi

# With the first replica running, deploy the rest unless we want to deploy only one replica
if [ $replicas -ne 1 ]; then
    kubectl scale -f ${deploymentFileName} -n $namespace --replicas=${replicas}
fi
</code></pre>
<p>I created simple example to illustrate you how it works.</p>
<p>First, I created the <code>web-app.yml</code> deployment manifest file:</p>
<pre><code>$ kubectl create deployment web-app --image=nginx --replicas=3 --dry-run=client -oyaml &gt; web-app.yml
</code></pre>
<p>Then I deployed the <code>web-app</code> Deployment using the <code>deploy.sh</code> script:</p>
<pre><code>$ ./deploy.sh web-app.yml default 3
deployment.apps/web-app created
Waiting for deployment &quot;web-app&quot; rollout to finish: 0 of 1 updated replicas are available...
deployment &quot;web-app&quot; successfully rolled out
deployment.apps/web-app scaled
</code></pre>
<p>From another terminal window, we can see that only when the first replica (<code>web-app-5cd54cb75-krgtc</code>) was in the &quot;Running&quot; state, the rest started to start:</p>
<pre><code>$ kubectl get pod -w
NAME                      READY   STATUS             RESTARTS   AGE
web-app-5cd54cb75-krgtc   0/1     Pending             0          0s
web-app-5cd54cb75-krgtc   0/1     Pending             0          0s
web-app-5cd54cb75-krgtc   0/1     ContainerCreating   0          0s
web-app-5cd54cb75-krgtc   1/1     Running             0          4s # First replica in the &quot;Running state&quot;
web-app-5cd54cb75-tmpcn   0/1     Pending             0          0s
web-app-5cd54cb75-tmpcn   0/1     Pending             0          0s
web-app-5cd54cb75-sstg6   0/1     Pending             0          0s
web-app-5cd54cb75-tmpcn   0/1     ContainerCreating   0          0s
web-app-5cd54cb75-sstg6   0/1     Pending             0          0s
web-app-5cd54cb75-sstg6   0/1     ContainerCreating   0          0s
web-app-5cd54cb75-tmpcn   1/1     Running             0          5s
web-app-5cd54cb75-sstg6   1/1     Running             0          7s
</code></pre>
<h3>Solution two: Using initContainers</h3>
<p>You can use the <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/" rel="nofollow noreferrer">init Container</a> that will run a script that determines which Pod should be run first:</p>
<pre><code>$ cat checker.sh
#!/bin/bash

labelName=&quot;app&quot; # label key of you rapplication
labelValue=&quot;web&quot; # label value of your application
hostname=$(hostname) 
apt update &amp;&amp; apt install -y jq 1&gt;/dev/null 2&gt;&amp;1 # install the jq program - command-line JSON processor
startFirst=$(curl -sSk -H &quot;Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)&quot; &quot;https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/pods/?labelSelector=${labelName}%3D${labelValue}&amp;limit=500&quot; | jq '.items[].metadata.name' | sort | head -n1 | tr -d '&quot;&quot;') # determine which Pod should start first -&gt; kubectl get pods -l app=web -o=name | sort | head -n1

firstPodStatusChecker=$(curl -sSk -H &quot;Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)&quot; &quot;https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/pods/${startFirst}&quot;| jq '.status.phase' | tr -d '&quot;&quot;') # check status of the Pod that should start first

attempts=0
if [ ${hostname} != ${startFirst} ]
then
    while [ ${firstPodStatusChecker} != &quot;Running&quot; ] &amp;&amp; [ $attempts -lt 60 ]; do
        attempts=$((attempts + 1))
        sleep 5
        firstPodStatusChecker=$(curl -sSk -H &quot;Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)&quot; &quot;https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/pods/${startFirst}&quot;| jq '.status.phase' | tr -d '&quot;&quot;') # check status of the Pod that should start first
    done
fi

if [ $attempts -eq 60 ]; then
    echo &quot;ERROR: Timeout&quot;
    exit 1
fi
</code></pre>
<p>The most important line in this script is:</p>
<pre><code>startFirst=$(curl -sSk -H &quot;Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)&quot; &quot;https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/pods/?labelSelector=${labelName}%3D${labelValue}&amp;limit=500&quot; | jq '.items[].metadata.name' | sort | head -n1 | tr -d '&quot;&quot;')
</code></pre>
<p>This line determines which Pod should start first and the rest of replicas will wait until the first Pod is started.
I'm using <code>curl</code> command to access the API from the Pod. We don't need to create complex <code>curl</code> commands manually but we can easily convert
<code>kubectl</code> command to <code>curl</code> command - if you run <code>kubectl</code> command with <code>-v=10</code> option you can see <code>curl</code> requests.</p>
<p><strong>NOTE:</strong> In this approach, you need to add appropriate permissions for the <code>ServiceAccount</code> to communicate with the API.
For example you may add  a <code>view</code> role to your <code>ServiceAccount</code> like this:</p>
<pre><code>$ kubectl create clusterrolebinding --serviceaccount=default:default --clusterrole=view default-sa-view-access
clusterrolebinding.rbac.authorization.k8s.io/default-sa-view-access created
</code></pre>
<p>You can mount this <code>checker.sh</code> script e.g. as a <code>ConfigMap</code>:</p>
<pre><code>$ cat check-script-configmap.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: check-script
data:
  checkScript.sh: |
    #!/bin/bash

    labelName=&quot;app&quot; # label key of you rapplication
    labelValue=&quot;web&quot; # label value of your application
    hostname=$(hostname) 
    apt update &amp;&amp; apt install -y jq 1&gt;/dev/null 2&gt;&amp;1 # install the jq program - command-line JSON processor
    startFirst=$(curl -sSk -H &quot;Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)&quot; &quot;https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/pods/?labelSelector=${labelName}%3D${labelValue}&amp;limit=500&quot; | jq '.items[].metadata.name' | sort | head -n1 | tr -d '&quot;&quot;') # determine which Pod should start first -&gt; kubectl get pods -l app=web -o=name | sort | head -n1

    firstPodStatusChecker=$(curl -sSk -H &quot;Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)&quot; &quot;https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/pods/${startFirst}&quot;| jq '.status.phase' | tr -d '&quot;&quot;') # check status of the Pod that should start first

    attempts=0  
    if [ ${hostname} != ${startFirst} ]
    then
        while [ ${firstPodStatusChecker} != &quot;Running&quot; ] &amp;&amp; [ $attempts -lt 60 ]; do
            attempts=$((attempts + 1))
            sleep 5
            firstPodStatusChecker=$(curl -sSk -H &quot;Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)&quot; &quot;https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/pods/${startFirst}&quot;| jq '.status.phase' | tr -d '&quot;&quot;') # check status of the Pod that should start first
        done
    fi

    if [ $attempts -eq 60 ]; then
        echo &quot;ERROR: Timeout&quot;
        exit 1
    fi
</code></pre>
<p>I also created simple example to illustrate you how it works.</p>
<p>First, I created the above <code>check-script</code> ConfigMap:</p>
<pre><code>$ kubectl apply -f check-script-configmap.yml
configmap/check-script created
</code></pre>
<p>Then I mounted this <code>ConfigMap</code> to the <code>initContainer</code> and deployed this Deployment:</p>
<pre><code>$ cat web.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: web
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      volumes:
        - name: check-script
          configMap:
            name: check-script
      initContainers:
      - image: nginx
        name: init
        command: [&quot;bash&quot;, &quot;/mnt/checkScript.sh&quot;]
        volumeMounts:
        - name: check-script
          mountPath: /mnt/

      containers:
      - image: nginx
        name: nginx

$ kubectl apply -f web.yml
deployment.apps/web created
</code></pre>
<p>From another terminal window, we can see that only when the first replica (<code>web-98c4d45dd-6zcsd</code>) was in the &quot;Running&quot; state, the rest started to start:</p>
<pre><code>$ kubectl get pod -w
NAME                  READY   STATUS    RESTARTS   AGE
web-98c4d45dd-ztjlf   0/1     Pending   0          0s
web-98c4d45dd-ztjlf   0/1     Pending   0          0s
web-98c4d45dd-6zcsd   0/1     Pending   0          0s
web-98c4d45dd-mc2ww   0/1     Pending   0          0s
web-98c4d45dd-6zcsd   0/1     Pending   0          0s
web-98c4d45dd-mc2ww   0/1     Pending   0          0s
web-98c4d45dd-ztjlf   0/1     Init:0/1   0          0s
web-98c4d45dd-6zcsd   0/1     Init:0/1   0          0s
web-98c4d45dd-mc2ww   0/1     Init:0/1   0          1s
web-98c4d45dd-6zcsd   0/1     Init:0/1   0          3s
web-98c4d45dd-ztjlf   0/1     Init:0/1   0          3s
web-98c4d45dd-mc2ww   0/1     Init:0/1   0          4s
web-98c4d45dd-6zcsd   0/1     PodInitializing   0          10s
web-98c4d45dd-6zcsd   1/1     Running           0          12s
web-98c4d45dd-mc2ww   0/1     PodInitializing   0          23s
web-98c4d45dd-ztjlf   0/1     PodInitializing   0          23s
web-98c4d45dd-mc2ww   1/1     Running           0          25s
web-98c4d45dd-ztjlf   1/1     Running           0          26s
</code></pre>
