<p>I want to display a maintenance page on an application running under Kubernetes whilst a deployment is in progress, in this “maintenance” window, I backup the database and then apply schema changes and then deploy the new version. </p>

<p>I thought maybe what I could do is change the service selector so that it would point to a nginx container serving up a simple maintenance page whilst the deployment progressed. Once the deployment had succeeded, I would switch back the selector to point to the pods that do the actual work.</p>

<p>My problem with this is approach is that unless I close and reopen the browser that is currently looking at the site then I never see the maintenance page; I’m guessing the browser is keeping a connection open. The public service address doesn’t change throughout this process.</p>

<p>I’m testing this locally on a Docker Kubernetes installation using a type of NodePort .</p>

<p>Any ideas on how to get it working or am I flogging a dead horse with this approach?</p>

<p>Regards
Lee</p>

LeeAlexander
<p>This happens due to a combination of how browsers and k8s services work.</p>

<p>Browsers cache TCP connections to servers: when requesting a page they will leave the TCP connection open, and if the user later requests more pages from the same domain, the browser will reuse the already-open TCP connection to save time.</p>

<p>The k8s service load balancing operates at the TCP layer. When a new TCP connection is received, it will be assigned to a pod from the Service, and it will keep talking to that pod for the entire TCP connection's lifetime.</p>

<p>So, the issue is your browser is keeping TCP connections open to your old pods, even if you modify the service.</p>

<p>How can we fix this?</p>

<p>Non-solution #1: have the browser not cache connections. As far as I know there's no way to do this, and you don't want it anyway because it'll make your site slower. Also, HTTP caching headers have no impact on this. Browsers always cache TCP connections. A no-cache header will make the browser request the page again, but over the already-open connection.</p>

<p>Non-solution #2: have k8s kill TCP connections when updating the service. This is not possible and is not desirable either because this behavior is what makes "graceful shutdown / request draining" deployment strategies work. See <a href="https://github.com/kubernetes/kubernetes/issues/46753" rel="nofollow noreferrer">issue</a>.</p>

<p>Solution #1: Use Layer 7 (HTTP) load balancing instead of Layer 4 (TCP) load balancing, such as <a href="https://github.com/kubernetes/ingress-nginx" rel="nofollow noreferrer">nginx-ingress</a>. L7 load balancing routes traffic to pods "per HTTP request", instead of "per TCP connection", so you won't have this problem even if browsers keep TCP connections open.</p>

<p>Solution #2: do this from your application instead of from k8s. For example, have an "in-maintenance" DB flag, check it on every request and serve the maintenance page if it's set.</p>
