<p>i have encountered a strange problem with my cluster</p>
<p>in my cluster i have a deployment and a Loadbalancer service exposing this deployment
it worked like a charm but suddenly the Loadbalancer started to return an error:</p>
<pre><code>curl: (56) Recv failure: Connection reset by peer
</code></pre>
<p>the error is showing while the pod and the loadbalancer are running and have no errors in their logs</p>
<p>what i already tried:</p>
<ul>
<li>deleting the pod</li>
<li>redeploying service + deployment from scratch
but the issue persist</li>
</ul>
<p>my service yaml:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/neg: '{&quot;ingress&quot;:true}'
    kubectl.kubernetes.io/last-applied-configuration: |
      {&quot;apiVersion&quot;:&quot;v1&quot;,&quot;kind&quot;:&quot;Service&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;labels&quot;:{&quot;app.kubernetes.io/instance&quot;:&quot;RELEASE-NAME&quot;,&quot;app.kubernetes.io/name&quot;:&quot;APP-NAME&quot;,&quot;app.kubernetes.io/version&quot;:&quot;latest&quot;},&quot;name&quot;:&quot;APP-NAME&quot;,&quot;namespace&quot;:&quot;namespacex&quot;},&quot;spec&quot;:{&quot;ports&quot;:[{&quot;name&quot;:&quot;web&quot;,&quot;port&quot;:3000}],&quot;selector&quot;:{&quot;app.kubernetes.io/instance&quot;:&quot;RELEASE-NAME&quot;,&quot;app.kubernetes.io/name&quot;:&quot;APP-NAME&quot;},&quot;type&quot;:&quot;LoadBalancer&quot;}}
  creationTimestamp: &quot;2021-08-03T07:55:00Z&quot;
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/name: APP-NAME
    app.kubernetes.io/version: latest
  name: APP-NAME
  namespace: namespacex
  resourceVersion: &quot;14583904&quot;
  uid: 7fb4d7e6-4316-44e5-8f9b-7a466bc776da
spec:
  clusterIP: 10.4.18.36
  clusterIPs:
  - 10.4.18.36
  externalTrafficPolicy: Cluster
  ports:
  - name: web
    nodePort: 30970
    port: 3000
    protocol: TCP
    targetPort: 3000
  selector:
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/name: APP-NAME
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: xx.xxx.xxx.xxx
</code></pre>
<p>my deployment yaml:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: APP-NAME
  labels:
    app.kubernetes.io/name: APP-NAME
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/version: &quot;latest&quot;
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: APP-NAME
      app.kubernetes.io/instance: RELEASE-NAME
  template:
    metadata:
      annotations:
        checksum/config: 5e6ff0d6fa64b90b0365e9f3939cefc0a619502b32564c4ff712067dbe44ab90
        checksum/secret: 76e0a1351da90c0cef06851e3aa9e7c80b415c29b11f473d4a2520ade9c892ce
      labels:
        app.kubernetes.io/name: APP-NAME
        app.kubernetes.io/instance: RELEASE-NAME
    spec:
      serviceAccountName: APP-NAME
      containers:
        - name: APP-NAME
          image: 'docker.io/xxxxxxxx:latest'
          imagePullPolicy: &quot;Always&quot;
          ports:
            - name: http
              containerPort: 3000
          livenessProbe:
            httpGet:
              path: /balancer/
              port: http
          readinessProbe:
            httpGet:
              path: /balancer/
              port: http
          env:
            ...
          volumeMounts:
            - name: config-volume
              mountPath: /home/app/config/
          resources:
            limits:
              cpu: 400m
              memory: 256Mi
            requests:
              cpu: 400m
              memory: 256Mi
      volumes:
        - name: config-volume
          configMap:
            name: app-config
      imagePullSecrets:
        - name: secret
</code></pre>

<p>The issue in my case turned to be a network component (like a FW) blocking the outbound connection after dimming the cluster 'unsafe' for no apparent reason</p>
<p>so in essence it was not a K8s issue but an IT one</p>
