<p>We want to dynamically scale our AKS Cluster based on the number of Websocket connections.</p>
<p>We use Application Gateway V2 along with Application Gateway Ingress Controller on AKS as Ingress.</p>
<p>I configured HorizontalPodAutoscaler to scale the deployment based on the consumed memory.</p>
<p>When i deploy the sample app to AKS i can connect to the websocket endpoints and communicate.
However, when any scale operation happens (pods added or removed) i see connection losses on all the clients.</p>
<ul>
<li>How can i keep the existing connections when pods are added?</li>
<li>How can i gracefully drain connections when pods are removed so existing clients are not affected?</li>
</ul>
<p>I tried activating cookie based affinity on application gateway but this had no effect on the issue.</p>
<p>Below is the deployment i use for testing. It is based on <a href="https://github.com/dotnet/AspNetCore.Docs/tree/main/aspnetcore/fundamentals/websockets/samples/6.x/WebSocketsSample" rel="nofollow noreferrer">this</a> sample and modified a but so it allows to specify the number of connections and regularily sends ping messages to the server.</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: wssample
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wssample
  template:
    metadata:
      labels:
        app: wssample
    spec:
      containers:
      - name: wssamplecontainer
        image: marxx/websocketssample:10
        resources:
          requests:
            memory: &quot;100Mi&quot;
            cpu: &quot;50m&quot;
          limits:
            memory: &quot;150Mi&quot;
            cpu: &quot;100m&quot;
        ports:
        - containerPort: 80
          name: wssample
---
apiVersion: v1
kind: Service
metadata:
  name: wssample-service
spec:
  ports:
  - port: 80
  selector:
    app: wssample
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: websocket-ingress
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/cookie-based-affinity: &quot;true&quot;
    appgw.ingress.kubernetes.io/connection-draining: &quot;true&quot;
    appgw.ingress.kubernetes.io/connection-draining-timeout: &quot;60&quot;
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name:  wssample-service
            port: 
              number: 80
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: websocket-scaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: wssample
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource: 
      name: memory
      target:
        type: Utilization
        averageUtilization: 50
</code></pre>
<p>Update:</p>
<ul>
<li>I am running on a 2-node cluster with autoscaler activated to scale up to 4 nodes.</li>
<li>There is still plenty of memory available on the nodes</li>
<li>At first i thought it was an issue with browsers and javascript but i got the same results when i connected to the endpoint via a .NET Core Based Console Application (Websockets went to state 'Aborted' after the scale operation)</li>
</ul>
<p>Update 2:</p>
<p>I found a pattern. The problem occurs also without HPA and can be reproduced using the following steps:</p>
<ol>
<li>Scale Deployment to 3 Replicas</li>
<li>Connect 20 Clients</li>
<li>Manually Scale Deployment to 6 Replicas with kubectl scale command</li>
<li>(existing connections are still fine and clients communicate with backend)</li>
<li>Connect another 20 Clients</li>
<li>After a few seconds all the existing connections are reset</li>
</ol>
<p>Update 3:</p>
<ul>
<li>The AKS cluster is using kubenet networking</li>
<li>Same issue with Azure CNI networking though</li>
</ul>

Markus S.
<p>I made a very unpleasant discovery. The outcome of <a href="https://github.com/Azure/application-gateway-kubernetes-ingress/issues/1277" rel="nofollow noreferrer">this GitHub issue</a> basically says that the behavior is by design and AGW resets all websocket connections when any backend pool rules change (which happens during scale operations).</p>
<p>It's possible to <a href="https://feedback.azure.com/d365community/idea/a2dcfc40-ba9f-ec11-a81c-000d3adfb8f5" rel="nofollow noreferrer">vote</a> for a feature to keep those connections in those situations.</p>
