<p>I've done quite a bit of searching and cannot seem to find anyone that shows a resolution to this problem.</p>
<p>I'm getting intermittent 111 Connection refused errors on my kubernetes clusters. It seems that about 90% of my requests succeed and the other 10% fail. If you &quot;refresh&quot; the page, a previously failed request will then succeed. I have 2 different Kubernetes clusters with the same exact setup both showing the errors.</p>
<p>This looks to be very close to what I am experiencing. I did install my setup onto a new cluster, but the same problem persisted:
<a href="https://stackoverflow.com/questions/58401610/kubernetes-clusterip-intermittent-502-connection-refused">Kubernetes ClusterIP intermittent 502 connection refused</a></p>
<p><strong>Setup</strong></p>
<ul>
<li>Kubernetes Cluster Version: 1.18.12-gke.1206</li>
<li>Django Version: 3.1.4</li>
<li>Helm to manage kubernetes charts</li>
</ul>
<p><strong>Cluster Setup</strong></p>
<p>Kubernetes nginx ingress controller that serves web traffic into the cluster:
<a href="https://kubernetes.github.io/ingress-nginx/deploy/#gce-gke" rel="nofollow noreferrer">https://kubernetes.github.io/ingress-nginx/deploy/#gce-gke</a></p>
<p>From there I have 2 Ingresses defined that route traffic based on the referrer url.</p>
<ol>
<li>Stage Ingress</li>
<li>Prod Ingress</li>
</ol>
<p><strong>Ingress</strong></p>
<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: potr-tms-ingress-{{ .Values.environment }}
  namespace: {{ .Values.environment }}
  labels:
    app: potr-tms-{{ .Values.environment }}
  annotations:
    kubernetes.io/ingress.class: &quot;nginx&quot;
    nginx.ingress.kubernetes.io/ssl-redirect: &quot;true&quot;
    nginx.ingress.kubernetes.io/from-to-www-redirect: &quot;true&quot;

# this line below doesn't seem to have an effect
#    nginx.ingress.kubernetes.io/service-upstream: &quot;true&quot;
    nginx.ingress.kubernetes.io/proxy-body-size: &quot;100M&quot;
    cert-manager.io/cluster-issuer: &quot;letsencrypt-{{ .Values.environment }}&quot;
spec:
  rules:
    - host: {{ .Values.ingress_host }}
      http:
        paths:
        - path: /
          backend:
            serviceName: potr-tms-service-{{ .Values.environment }}
            servicePort: 8000
  tls:
    - hosts:
      - {{ .Values.ingress_host }}
      - www.{{ .Values.ingress_host }}
      secretName: potr-tms-{{ .Values.environment }}-tls
</code></pre>
<p>These ingresses route to 2 services that I have defined for prod and stage:</p>
<p><strong>Service</strong></p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: potr-tms-service-{{ .Values.environment }}
  namespace: {{ .Values.environment }}
  labels:
    app: potr-tms-{{ .Values.environment }}
spec:
  type: ClusterIP
  ports:
    - name: potr-tms-service-{{ .Values.environment }}
      port: 8000
      protocol: TCP
      targetPort: 8000
  selector:
    app: potr-tms-{{ .Values.environment }}
</code></pre>
<p>These 2 services route to deployments that I have for both prod and stage:</p>
<p><strong>Deployment</strong></p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: potr-tms-deployment-{{ .Values.environment }}
  namespace: {{ .Values.environment }}
  labels:
    app: potr-tms-{{ .Values.environment }}
spec:
  replicas: {{ .Values.deployment_replicas }}
  selector:
    matchLabels:
      app: potr-tms-{{ .Values.environment }}
  strategy:
    type: RollingUpdate
  template:
    metadata:
      annotations:
        rollme: {{ randAlphaNum 5 | quote }}
      labels:
        app: potr-tms-{{ .Values.environment }}
    spec:
      containers:
      - command: [&quot;gunicorn&quot;, &quot;--bind&quot;, &quot;:8000&quot;, &quot;config.wsgi&quot;]
#      - command: [&quot;python&quot;, &quot;manage.py&quot;, &quot;runserver&quot;, &quot;0.0.0.0:8000&quot;]
        envFrom:
          - secretRef:
              name: potr-tms-secrets-{{ .Values.environment }}
        image: gcr.io/potrtms/potr-tms-{{ .Values.environment }}:latest
        name: potr-tms-{{ .Values.environment }}
        ports:
        - containerPort: 8000
        resources:
          requests:
            cpu: 200m
            memory: 512Mi
      restartPolicy: Always
      serviceAccountName: &quot;potr-tms-service-account-{{ .Values.environment }}&quot;
status: {}
</code></pre>
<p><strong>Error</strong>
This is the error that I'm seeing inside of my ingress controller logs:
<a href="https://i.stack.imgur.com/mRAki.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mRAki.png" alt="Error log" /></a></p>
<p>This seems pretty clear, if my deployment pods were failing or showing errors they would be &quot;unavailable&quot; and the service would not be able to route them to the pod. To try and debug this I did increase my deployment resources and replica counts. The amount of web traffic to this app is pretty low though, ~10 users.</p>
<p><strong>What I've Tried</strong></p>
<ol>
<li>I tried using a completely different ingress controller <a href="https://github.com/kubernetes/ingress-nginx" rel="nofollow noreferrer">https://github.com/kubernetes/ingress-nginx</a></li>
<li>Increasing deployment resources / replica counts (seems to have no effect)</li>
<li>Installing my whole setup on a brand new cluster (same results)</li>
<li>restart the ingress controller / deleting and re installing</li>
<li>Potentially it sounds like this could be a Gunicorn problem. To test I tried starting my pods with python manage.py runserver, problem remained.</li>
</ol>
<p><strong>Update</strong></p>
<p>Raising the pod counts seems to have helped a little bit.</p>
<ul>
<li>deployment replicas: 15</li>
<li>cpu request: 200m</li>
<li>memory request: 512Mi</li>
</ul>
<p>Some requests do fail still though.</p>

<p>Did you find a solution to this?  I am seeing something very similar on a minikube setup.</p>
<p>In my case, I believe I also see the nginx controller restarting after the 502.  The 502 is intermittent, frequently the first access fails, then reload works.</p>
<p>The best idea I've found so far is to increase the Nginx timeout parameter, but I have not tried that yet.  Still trying to search out all options.</p>
