<p>I'm trying to find an optimal way to handle ongoing PostgreSQL transactions during the shutdown of a golang server running on Kubernetes.</p>
<p>Does it make sense to wait for transactions to finish, when these transaction are serving requests initiated by a server that has already shutdown? And even if the transaction completes within the graceful shutdown timeout duration - will the server be able to send the response?</p>
<p>Even if responding to ongoing requests during shutdown is not possible, I prefer to cancel the context of all running transaction so they don't continue to run on the database after the server terminates, adding unnecessary load. But whenever I wait for transactions to finish, it seems there's a trade-off: The longer I wait for ongoing transactions to finish - the longer the container exists with a non responsive server that would error on each incoming request.</p>
<p>Here's some sample code that demonstrates this:</p>
<pre class="lang-golang prettyprint-override"><code>import (
    &quot;github.com/jackc/pgx/v5/pgxpool&quot;
    &quot;os/signal&quot;
    &quot;context&quot;
    &quot;net/http&quot;
    &quot;syscall&quot;
    &quot;time&quot;
)

func main() {
    ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGQUIT, syscall.SIGINT)
    defer cancel()

    // db is used by the API handler functions
    db, err := pgxpool.NewWithConfig(ctx, &lt;some_config&gt;)
    if err != nil {
        logger.Error(&quot;server failed to Shutdown&quot;, err)
    }


    server := http.Server{&lt;some_values&gt;}
    serverErr := make(chan error)
    go func() {
        serverErr &lt;- server.ListenAndServe()
    }()

    select {
    case &lt;-ctx.Done():
        if err := Shutdown(closeCtx, time.Second*10, server, db); err != nil {
            logger.Error(&quot;server failed to Shutdown&quot;, err)
        }

    case err := &lt;-serverErr:
        logger.Error(&quot;server failed to ListenAndServe&quot;, err)
    }
}

func Shutdown(ctx context.Context, timeout time.Duration, server *http.Server, db *pgxpool.Pool) error {

    closeCtx, cancel := context.WithTimeout(ctx, timeout)
    defer cancel()

    // first, shutdown the server to stop accepting new requests
    if err := server.Shutdown(closeCtx); err != nil {
        return err
    }

    // allow running transactions to finish, but if they don't finish within
    // ten seconds, cancel the context of all running transactions so that they
    // are forced to finish (albeit, with error)
    transactionsComplete := waitForTransacitons(time.Second*10, db)
    if !transactionsComplete {
        cancelContextOfEveryTransaction()
    }
    
    // since this call blocks until all transactions finished we must call it
    // only once we are sure that there are no more running transactions.
    db.Close(ctx)

    return nil
}
</code></pre>
<p>Would the optimal graceful termination sequence be:</p>
<ul>
<li>Shutdown the server.</li>
<li>Immediately cancel context of all ongoing requests (killing the transaction as soon as the database driver tries to do anything with it).</li>
<li>Close the connection pool.</li>
<li>Exit.</li>
</ul>
<p>[edit]: alternative termination sequence (more graceful):</p>
<ul>
<li>Termination signal is received.</li>
<li>The pod is in 'terminating' state and is removed from the load balancer.</li>
<li>Shutdown the server with some timeout N.</li>
<li>Shutdown the connection pool - with a short timeout. <em>Reasoning: since <code>server.Shutdown</code> returned, no responses will be returned. The only reason to wait for ongoing transactions is for background workers to finish their work, such as writing logs to the database.</em></li>
<li>If there are still open transaction that prevent the connection pool from closing - kill these transactions and try to close the pool again.</li>
<li>Exit.</li>
</ul>

Alechko
<p>Why reinventing the wheel and not using some of the existing libraries, that do the magic for you?</p>
<p>In our production services, we have used <a href="https://github.com/TV4/graceful" rel="nofollow noreferrer">this graceful shutdown lib</a> a lot and never had issues with it. It waits until all HTTP requests are served (within given timeout) and shuts down afterwards.</p>
<p>The usage couldn't be simpler. After installing it</p>
<pre><code>go mod download github.com/TV4/graceful
</code></pre>
<p>(eventually:</p>
<pre><code>go get -u github.com/TV4/graceful
</code></pre>
<p>)</p>
<p>you only need to import it:</p>
<pre><code>import (
    // ...

    &quot;github.com/TV4/graceful&quot;
)
</code></pre>
<p>and then you can replace all your code after instantiating a <code>server</code> (including your Shutdown function) with this one-liner:</p>
<pre><code>server := ...
graceful.LogListenAndServe(server, logger)
</code></pre>
