<p>I'm maintaining a Kubernetes cluster which includes two PostgreSQL servers in two different pods, a primary and a replica.  The replica is sync'ed from the primary via log shipping.</p>
<p>A glitch caused the log shipping to start failing so the replica is no longer in sync with the primary.</p>
<p>The process for bringing a replica back into sync with the primary requires, amongst other things, stopping the postgres service of the replica.  And this is where I'm having trouble.</p>
<p>It appears that Kubernetes is restarting the container as soon as I shut down the postgres service, which immediately restarts postgres again.  I need the container running with the postgres service inside it stopped, to allow me to perform the next steps in fixing the broken replication.</p>
<p>How can I get Kubernetes to allow me to shut down the postgres service without restarting the container?</p>
<p><strong>Further Details:</strong></p>
<p>To stop the replica I run a shell on the replica pod via <code>kubectl exec -it &lt;pod name&gt; -- /bin/sh</code>, then run <code>pg_ctl stop</code> from the shell.  I get the following response:</p>
<pre><code>server shutting down
command terminated with exit code 137
</code></pre>
<p>and I'm kicked out of the shell.</p>
<p>When I run <code>kubectl describe pod</code> I see the following:</p>
<pre><code>Name:         pgset-primary-1
Namespace:    qa
Priority:     0
Node:         aks-nodepool1-95718424-0/10.240.0.4
Start Time:   Fri, 09 Jul 2021 13:48:06 +1200
Labels:       app=pgset-primary
              controller-revision-hash=pgset-primary-6d7d65c8c7
              name=pgset-replica
              statefulset.kubernetes.io/pod-name=pgset-primary-1
Annotations:  &lt;none&gt;
Status:       Running
IP:           10.244.1.42
IPs:
  IP:           10.244.1.42
Controlled By:  StatefulSet/pgset-primary
Containers:
  pgset-primary:
    Container ID:   containerd://bc00b4904ab683d9495ad020328b5033ecb00d19c9e5b12d22de18f828918455
    Image:          *****/crunchy-postgres:centos7-9.6.8-1.6.0
    Image ID:       docker.io/*****/crunchy-postgres@sha256:2850e00f9a619ff4bb6ff889df9bcb2529524ca8110607e4a7d9e36d00879057
    Port:           5432/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sat, 06 Nov 2021 18:29:34 +1300
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 06 Nov 2021 18:28:09 +1300
      Finished:     Sat, 06 Nov 2021 18:29:18 +1300
    Ready:          True
    Restart Count:  6
    Limits:
      cpu:     250m
      memory:  512Mi
    Requests:
      cpu:     10m
      memory:  256Mi
    Environment:
      PGHOST:                 /tmp
      PG_PRIMARY_USER:        primaryuser
      PG_MODE:                set
      PG_PRIMARY_HOST:        pgset-primary
      PG_REPLICA_HOST:        pgset-replica
      PG_PRIMARY_PORT:        5432
      [...]
      ARCHIVE_TIMEOUT:        60
      MAX_WAL_KEEP_SEGMENTS:  400
    Mounts:
      /backrestrepo from backrestrepo (rw)
      /pgconf from pgbackrestconf (rw)
      /pgdata from pgdata (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from pgset-sa-token-nh6ng (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  pgdata:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pgdata-pgset-primary-1
    ReadOnly:   false
  backrestrepo:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  backrestrepo-pgset-primary-1
    ReadOnly:   false
  pgbackrestconf:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pgbackrest-configmap
    Optional:  false
  pgset-sa-token-nh6ng:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  pgset-sa-token-nh6ng
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                 From     Message
  ----     ------   ----                ----     -------
  Warning  BackOff  88m (x3 over 3h1m)  kubelet  Back-off restarting failed container
  Normal   Pulled   88m (x7 over 120d)  kubelet  Container image &quot;*****/crunchy-postgres:centos7-9.6.8-1.6.0&quot; already present on machine
  Normal   Created  88m (x7 over 120d)  kubelet  Created container pgset-primary
  Normal   Started  88m (x7 over 120d)  kubelet  Started container pgset-primary
</code></pre>
<p>The events suggest the container was started by Kubernetes.</p>
<p>The pod has no liveness or readiness probes so I don't know what would prompt Kubernetes to restart the container when I shut down the postgres service running within it.</p>

Simon Elms
<p>This happens due to <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy" rel="nofollow noreferrer">restartPolicy</a>. Container lifecycle is terminated due to its process being completed. If you do not want a new container to be created you need to change the restart policy for these pods.</p>
<p>If this pod is a part of deployment, look into <code>kubectl explain deployment.spec.template.spec.restartPolicy</code></p>
