<p>I have a Kubernetes cluster with an nginx ingress controller. The nginx controller often randomly reloads
<a href="https://i.stack.imgur.com/KXTLg.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KXTLg.png" alt="enter image description here" /></a></p>
<p>On this cluster i've now deployed a web socket server. And it seems like every time the nginx ingress controller restarts, the web socket connection also gets closed. I can't seem to find out why the nginx ingress controller is restarting and how I can prevent this from happening or can prevent the web socket from closing the connection.</p>
<p>Does anyone know what I can do to keep the web socket connections alive?</p>
<p>Cloud provider: Google Kubernetes Engine</p>
<p>Kubernetes Version: 1.18.20-gke.501</p>
<p>Application Package Manager: Helm v3</p>
<p>YAML of the nginx-ingress-controller (We haven't changed anything in this YAML, I reckon it's a default configuration):</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  creationTimestamp: &quot;2018-06-19T14:21:41Z&quot;
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app: nginx-ingress
    chart: nginx-ingress-0.12.2
    component: controller
    heritage: Tiller
    release: nginx-ingress-1
  name: nginx-ingress-1-controller
  namespace: loadbalancers
  resourceVersion: &quot;277233411&quot;
  selfLink: /api/v1/namespaces/loadbalancers/services/nginx-ingress-1-controller
  uid: 278d0e86-73cc-22e8-4567-35725aa57fef
spec:
  clusterIP: (ip-goes-here)
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    nodePort: 32269
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    nodePort: 32258
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx-ingress
    component: controller
    release: nginx-ingress-1
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: (ip-goes-here)
</code></pre>
<p>Websocket deployment:</p>
<p>deployment.yaml</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ template &quot;websocket-server.fullname&quot; . }}
  labels:
    app: {{ template &quot;websocket-server.name&quot; . }}
    chart: {{ template &quot;websocket-server.chart&quot; . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ template &quot;websocket-server.name&quot; . }}
      release: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ template &quot;websocket-server.name&quot; . }}
        release: {{ .Release.Name }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          envFrom:
            - configMapRef:
                name: env-vars
          image: &quot;{{ .Values.image.repository }}:{{ .Values.image.tag }}&quot;
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: 8001
              protocol: TCP
          resources:
{{ toYaml .Values.resources | indent 12 }}
    {{- with .Values.nodeSelector }}
      nodeSelector:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.affinity }}
      affinity:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
{{ toYaml . | indent 8 }}
    {{- end }}

      # Get credentials of the docker
      # registry from secret
      imagePullSecrets:
      - name: {{ .Values.image.imagePullSecrets }}
</code></pre>
<p>values.yaml:</p>
<pre><code>replicaCount: 1

image:
  repository: eu.gcr.io/test-234231/websocket-server
  tag: latest
  pullPolicy: Always
  imagePullSecrets: test

service:
  type: ClusterIP
  port: 8001

ingress:
  enabled: true
  path: /
  hosts:
    - ws.domainhere.com
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: &quot;true&quot;
  tls:
    - secretName: tls-certificate
      hosts:
        - ws.domainhere.com

resources: 
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  limits:
   cpu: 100m
   memory: 150Mi
  requests:
   cpu: 50m
   memory: 100Mi

nodeSelector: {}

tolerations: []

affinity: {}

envVars:
</code></pre>
<p>ingress.yaml</p>
<pre><code>{{- if .Values.ingress.enabled -}}
{{- $fullName := include &quot;websocket-server.fullname&quot; . -}}
{{- $ingressPath := .Values.ingress.path -}}
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: {{ $fullName }}
  labels:
    app: {{ template &quot;websocket-server.name&quot; . }}
    chart: {{ template &quot;websocket-server.chart&quot; . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
{{- with .Values.ingress.annotations }}
  annotations:
{{ toYaml . | indent 4 }}
{{- end }}
spec:
{{- if .Values.ingress.tls }}
  tls:
  {{- range .Values.ingress.tls }}
    - hosts:
      {{- range .hosts }}
        - {{ . }}
      {{- end }}
      secretName: {{ .secretName }}
  {{- end }}
{{- end }}
  rules:
  {{- range .Values.ingress.hosts }}
    - host: {{ . }}
      http:
        paths:
          - path: {{ $ingressPath }}
            backend:
              serviceName: {{ $fullName }}
              servicePort: http
  {{- end }}
{{- end }}
</code></pre>
<p>Dockerfile</p>
<pre><code>FROM python:3.8

EXPOSE 8001

ENV PYTHONUNBUFFERED 1

RUN groupadd -r tornado \
    &amp;&amp; useradd -r -g tornado tornado

# Requirements have to be pulled and installed here, otherwise caching won't work
COPY ./requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt &amp;&amp; rm -rf /requirements

COPY . /app

RUN chown -R tornado /app

USER tornado

WORKDIR /app

ENTRYPOINT [&quot;python3&quot;, &quot;asyncio_server.py&quot;]
</code></pre>
<p>I've tried setting keep-alive to &quot;0&quot; in the ConfigMap of the loadbalancer on staging. After this however the issue still presists.
The logs look as following:</p>
<pre><code>I0817 09:41:54.852526       6 event.go:218] Event(v1.ObjectReference{Kind:&quot;ConfigMap&quot;, Namespace:&quot;loadbalancers&quot;, Name:&quot;nginx-ingress-1-controller&quot;, UID:&quot;278d0e86-73cc-22e8-4567-35725aa57fef&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;463499015&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'UPDATE' ConfigMap loadbalancers/nginx-ingress-1-controller
I0817 09:41:54.922325       6 controller.go:183] backend reload required
I0817 09:41:55.578489       6 controller.go:192] ingress backend successfully reloaded...
I0817 09:44:16.384627       6 controller.go:183] backend reload required
I0817 09:44:17.134365       6 controller.go:192] ingress backend successfully reloaded...
I0817 09:49:08.813187       6 controller.go:183] backend reload required
I0817 09:49:09.478537       6 controller.go:192] ingress backend successfully reloaded...
</code></pre>
<p>In another environment where we get more traffic the logs look like this:</p>
<pre><code>I0817 09:52:46.314745       6 controller.go:183] backend reload required
I0817 09:52:48.656831       6 controller.go:192] ingress backend successfully reloaded...
I0817 09:52:48.664464       6 controller.go:183] backend reload required
2021/08/17 09:52:49 [error] 550545#550545: *4035841 connect() failed (111: Connection refused) while connecting to upstream, client: 10.4.3.1, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
10.4.3.1 - [10.4.3.1] - - [17/Aug/2021:09:52:49 +0000] &quot;POST host HTTP/1.1&quot; 200 12 &quot;-&quot; &quot;&quot; 1583 0.027 [app-80] xx.xxx.x.xx:5000, xx.xxx.x.xx:5000 0, xx.xxx.x.xx, 0.026 502, 200
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035848 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
10.4.3.1 - [10.4.3.1] - - [17/Aug/2021:09:52:50 +0000] &quot;POST host HTTP/1.1&quot; 200 12 &quot;-&quot; &quot;&quot; 1588 0.107 [app-80] xx.xxx.x.xx:5000 xx.xxx.x.xx 200
xx.xxx.x.xx - [xx.xxx.x.xx] - - [17/Aug/2021:09:52:50 +0000] &quot;POST host HTTP/1.1&quot; 200 12 &quot;-&quot; &quot;&quot; 1644 0.155 [app-80] xx.xxx.x.xx:5000, xx.xxx.x.xx:5000, xx.xxx.x.xx:5000, xx.xxx.x.xx:5000, xx.xxx.x.xx:5000, 502, 502, 502, 502, 502, 502, 502, 502, 502, 502, 502, 502, 502, 502, 502, 502, 502, 200
2021/08/17 09:52:50 [error] 550545#550545: *4035867 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035867 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035867 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035867 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035867 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035867 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035867 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035867 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035867 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
2021/08/17 09:52:50 [error] 550545#550545: *4035867 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.x.xx, server: client.company.com, request: &quot;POST host HTTP/1.1&quot;, upstream: &quot;host&quot;, host: &quot;client.company.com&quot;
xx.xx.xxx.xx - [xx.xx.xxx.xx] - - [17/Aug/2021:09:52:50 +0000] &quot;POST host HTTP/1.1&quot; 200 12 &quot;-&quot; &quot;&quot; 1527 0.099 [app-80] xx.xxx.x.xx:5000, xx.xxx.x.xx:5000, xx.xxx.x.xx:5000, xx.xxx.x.xx:5000 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 12 0.000, 0.001, 0.006, 0.002, 0.001, 0.001, 0.004, 0.001, 0.001, 0.001, 0.081 502, 502, 502, 502, 502, 502, 502, 502, 502, 502, 200
xx.xx.xxx.xx - [xx.xx.xxx.xx] - - [17/Aug/2021:09:52:50 +0000] &quot;GET /?user_preferred_language=pl HTTP/2.0&quot; 200 605 &quot;host&quot; &quot;Mozilla/5.0 (Linux; Android 11; SAMSUNG SM-A025G) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/14.2 Chrome/87.0.4280.141 Mobile Safari/537.36&quot; 408 0.002 [app-http] xx.xxx.x.xx:80 1085 0.002 200
xx.xx.xxx.xx - [xx.xx.xxx.xx] - - [17/Aug/2021:09:52:50 +0000] &quot;GET /api HTTP/1.1&quot; 301 0 &quot;-&quot; &quot;python-requests/2.21.0&quot; 266 0.002 [second-app-80] xx.x.x.xx:5000 0 0.002 301
xx.xx.xxx.xx - [xx.xx.xxx.xx] - - [17/Aug/2021:09:52:50 +0000] &quot;GET /api HTTP/1.1&quot; 301 0 &quot;-&quot; &quot;python-requests/2.21.0&quot; 266 0.003 [second-app-80] xx.x.x.xx:5000 0 0.003 301
xx.xx.xxx.xx - [xx.xx.xxx.xx] - - [17/Aug/2021:09:52:50 +0000] &quot;GET /api HTTP/1.1&quot; 200 231 &quot;-&quot; &quot;python-requests/2.21.0&quot; 267 0.037 [second-app-80] xx.x.x.xx:5000 277 0.037 200
xx.xx.xxx.xx - [xx.xx.xxx.xx] - - [17/Aug/2021:09:52:50 +0000] &quot;GET /api HTTP/1.1&quot; 200 231 &quot;-&quot; &quot;python-requests/2.21.0&quot; 267 0.039 [second-app-80] xx.x.x.xxx:5000 277 0.039 200
I0817 09:52:51.752194       6 controller.go:192] ingress backend successfully reloaded...
</code></pre>

Jeroen Beljaars
<p>Whenever you create a new ingress with ingress-nginx, a new location will be added to Nginx config file in the ingress-nginx pod. So whenever there is a change in Nginx config file all nginx workers will restart so new config file can be used. Old workers will serve existing requests for some time (worker_restart_timeout) and all new requests will go to new workers. after worker_restart_timeout old workers will stop and your WebSocket connection served by that worker will be dropped.</p>
<p>worker_restart_timeout can be increased so your existing connection won't be closed for that time but if there are so many changes in the Nginx config file old workers will be in shutting down state for that time and new worker will be created. Eventually, your pod will be evicted with OOM error.</p>
