<p>I have a small hobby website I want to release on my server. I chose to use kubernetes for that, as I partly use it at work as well, so I'd like to get familiar. I bought hetzner debian server and installed k3s. Now are deploying PostgreSQL container (version 15.2, not cluster) as per this <a href="https://www.containiq.com/post/deploy-postgres-on-kubernetes" rel="nofollow noreferrer">tutorial</a>. (I did a minor changes I saw from other tutorials, should not be relevant).</p>
<p>It runs fine, I was happy with it. But I tried to restart deployment to make sure data are not lost if server goes down for some reason. After a few restarts, database is corrupted.</p>
<p>Once I saw:</p>
<pre><code>PANIC:  invalid magic number 0000 in log segment 000000010000000000000000, offset 0
</code></pre>
<p>another time:</p>
<pre><code>invalid contrecord length 1174 (expected 48430224) at 0/195BC90
</code></pre>
<p>another time:</p>
<pre><code>PANIC:  could not locate a valid checkpoint record
</code></pre>
<p>When I tried to google how to recover from this, I did not find any safe options and mostly the suggestions were to restore backup.</p>
<p>So my question is, how do I safely restart/shutdown PostgreSQL container? Am I missing some shutdown config for PostgreSQL pod in k8s?</p>
<p>Update 1:</p>
<p>I was restarting deployment from k9s with <code>r</code> command. I think UI made it look like it was rotated right away, but it probably takes some time. So I think I triggered multiple restarts every 10 seconds and that might have corrupted the DB. Anyway I added <code>terminationGracePeriodSeconds: 60</code> and used <code>preStop</code> hook from answer. Thanks</p>
<p>Update 2:
I imported DB, did a restart and again same issue:</p>
<blockquote>
<p>could not locate a valid checkpoint record</p>
</blockquote>
<p>Update 3:
<strong>I replaced <code>Deployment</code> with <code>StatefulSet</code> and it seems to be handling restarts better. Tried over 10 restarts and no issues. Whereas before it crashed around 4th restart.</strong></p>

Lucas03
<p>Of course the best practice is using an operator like <a href="https://cloudnative-pg.io/" rel="nofollow noreferrer">cloudnative-pg</a> or <a href="https://postgres-operator.readthedocs.io/en/latest/" rel="nofollow noreferrer">postgres-operator</a> but they are pretty big and probably have way more feature for a simple workload. Here is a simple solution for your problem.</p>
<h2>Solution</h2>
<p>Add below to your pod spec</p>
<pre class="lang-yaml prettyprint-override"><code>preStop:
  exec:
    command: [&quot;/usr/local/bin/pg_ctl stop -D /var/lib/postgresql/data -w -t 60 -m fast&quot;]
</code></pre>
<h2>Explanation</h2>
<p>Basically when you kill a pod, Kubernetes signals <code>SIGTERM</code> and gives 30 seconds for your pod, after that time it sends <code>SIGKILL</code>. When postgres receive <code>SIGTERM</code> it won't accept net connections but it won't terminate existing terminations, so any client will block db's termination, and after 30 seconds pod will receive <code>SIGKILL</code> which is very bad for postgres <a href="https://www.postgresql.org/docs/current/server-shutdown.html" rel="nofollow noreferrer">doc</a>. So you need to safely shutdown postgres somehow, with <code>preStop</code> hook you can.</p>
<h3>Kubernetes</h3>
<p>This is the exact chronological order of your pod:</p>
<ol>
<li>Set <code>state=Terminating</code> from Pod controller</li>
<li><code>terminationGracePeriodSeconds</code> timer starts (default is 30 seconds)</li>
<li><code>preStop</code> hook: <code>pg_cli ...</code></li>
<li><code>SIGTERM</code> is sent: Postgres won't accept new connections</li>
<li>k8s waits until <code>terminationGracePeriods</code> (configurable from yaml)</li>
<li>If app is still alive <code>SIGKILL</code> is sent</li>
</ol>
<p>Also you need to set <code>.spec.strategy.type==Recreate</code> in Deployment.</p>
<h2>Postgres</h2>
<p>For the <code>pg_cli</code> commands you can refer this summary, most useful one for you looks like <code>-m fast</code>.</p>
<p><code>SIGTERM</code>:</p>
<ul>
<li>&quot;Smart Shutdown Mode&quot;</li>
<li>Disallows new connections</li>
<li>Let's existing connections continue</li>
</ul>
<p><code>SIGINT</code>:</p>
<ul>
<li>&quot;Fast Shutdown Mode&quot;</li>
<li>Disallow new connections</li>
<li>Send <code>SIGTERM</code> to existing servers processes (?) they'll exit promptly</li>
</ul>
<p><code>SIGQUIT</code>:</p>
<ul>
<li>&quot;Immediate Shutdown Mode&quot;</li>
<li>Send <code>SIGQUIT</code> to all child processes, if they don't terminate in 5 secs sends <code>SIGKILL</code></li>
</ul>
<p>EDIT:</p>
<p>Apparently <code>Recreate</code> only guarantees <em>recreation</em> for update (old rs to new rs), but it <strong>does not guarantee 1 pod at a time</strong> if pod randomly dies. While new pod is creating old one may be in terminating phase, and because of race condition data may be corrupt. <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#recreate-deployment" rel="nofollow noreferrer">Relevant Doc</a></p>
<blockquote>
<p>This will only guarantee Pod termination previous to creation for upgrades. If you upgrade a Deployment, all Pods of the old revision will be terminated immediately. Successful removal is awaited before any Pod of the new revision is created. If you manually delete a Pod, the lifecycle is controlled by the ReplicaSet and the replacement will be created immediately (even if the old Pod is still in a Terminating state). <strong>If you need an &quot;at most&quot; guarantee for your Pods, you should consider using a StatefulSet.</strong></p>
</blockquote>
