<p>My (GCE) Load Balancer health checks are failing with a <code>connection refused</code> error, ultimately marking my GCE Ingress as <code>UNHEALTHY</code>.
Now I'm wondering how to fix this issue.</p>
<p>For my setup I'm using a <strong>GKE Autopilot</strong> cluster. And I have teared down and restarted my setup several times, always leading to the same result.</p>
<p>Suppose I have a deployment configured with a pod template consisting of several containers (of which not all of them expose ports).</p>
<p><strong>Side Note</strong>: For simplicity I skipped some configurations, such as config maps.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: some-app-deployment
spec:
  selector:
    matchLabels:
      app: some-app
  replicas: 2
  template:
    metadata:
      labels:
        app: some-app
    spec:
      restartPolicy: Always
      containers:
        - name: web-server
          image: {{some-app-image}}
          command: ['app', 'web-server']
          ports:
            - name: web-server
              containerPort: 5000
              protocol: TCP
        - name: admin-server
          image: {{some-app-image}}
          command: ['app', 'admin-server']
          ports:
            - name: admin-server
              containerPort: 5001
              protocol: TCP
        - name: worker
          image: {{some-app-image}}
          command: ['app', 'worker']
        - name: cron
          image: {{some-app-image}}
          command: ['app', 'cron']
        - name: helper
          image: {{some-app-image}}
          command: [ '/bin/bash', '-c', '--' ]
          args: [ 'while true; do sleep 30; done;' ]
</code></pre>
<p>The following is the BackendConfig CRD, which supposedly defines the health check.
I chose to define the path as <code>/favicon.ico</code> because Load Balancer Health Check requires exactly <code>200 OK</code> response and the web-servers base path <code>/</code> emits a redirect <code>302</code>, hence it would faild the Health Check.
With <code>kubectl port-forward</code> I confirmed that <code>/favicon.ico</code> actually emits <code>200 OK</code> and it does.
Btw. just to exclude this from being a problem, I also tried other paths without success.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: http-hc-config
spec:
  healthCheck:
    type: HTTP
    port: 5000
    requestPath: &quot;/favicon.ico&quot;
    checkIntervalSec: 20
</code></pre>
<p>Additionally there is a custom header added to the <code>admin</code> endpoint.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: x-header-config
spec:
  customRequestHeaders:
    headers:
    - &quot;X-Client-Region:{client_region}&quot;
</code></pre>
<p>The following is the service description. It references the BackendConfig via annotations as per the documentation.
The documentation is not very specific as to how to reference the health check, so I mapped it to the relevant port.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  name: some-app-service
  annotations:
    cloud.google.com/backend-config: '{
      &quot;default&quot;:&quot;http-hc-config&quot;,
      &quot;ports&quot;:{&quot;4001&quot;:&quot;x-header-config&quot;}}'
spec:
  type: NodePort
  selector:
    app: some-app
  ports:
    - name: web
      targetPort: 5000
      protocol: TCP
      port: 4000
    - name: admin
      targetPort: 5001
      protocol: TCP
      port: 4001
</code></pre>
<p>I confirmed that my service is running, as I could successfully <code>kubectl port-forward</code> the pods and access the web-servers content.</p>
<p>Now the final piece of the setup is this ingress object.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: gce
spec:
  rules:
    - host: admin.example.com
      http:
        paths:
        - path: /*
          pathType: ImplementationSpecific
          backend:
            service:
              name: some-app-service
              port:
                number: 4001
    - http:
        paths:
        - path: /*
          pathType: ImplementationSpecific
          backend:
            service:
              name: some-app-service
              port:
                number: 4000
</code></pre>
<p>EDIT #1 &amp; #2:
Performing a <code>gcloud compute health-checks describe {{HEALTH_DEF_ID}}</code> I receive the following output for the health check on <code>port 4001</code>, which seems to be out-of-sync with the defined BackendConfig CRD:</p>
<pre class="lang-yaml prettyprint-override"><code>checkIntervalSec: 15
creationTimestamp: '2022-02-07T11:30:50.059-08:00'
description: Default kubernetes L7 Loadbalancing health check for NEG.
healthyThreshold: 1
httpHealthCheck:
  portSpecification: USE_SERVING_PORT
  proxyHeader: NONE
  requestPath: /
id: {{REDACTED}}
kind: compute#healthCheck
logConfig:
  enable: true
name: k8s1-4622dadc-{{REDACTED}}
selfLink: https://www.googleapis.com/compute/v1/projects/{{REDACTED}}
timeoutSec: 15
type: HTTP
unhealthyThreshold: 2
</code></pre>
<p>And the following for <code>port 4000</code>, which surprisingly contains the right <code>path</code> and <code>port</code> configuration:</p>
<pre class="lang-yaml prettyprint-override"><code>checkIntervalSec: 20
creationTimestamp: '2022-02-07T12:12:59.248-08:00'
description: Default kubernetes L7 Loadbalancing health check for NEG.
healthyThreshold: 1
httpHealthCheck:
  port: 5000
  portSpecification: USE_FIXED_PORT
  proxyHeader: NONE
  requestPath: /favicon.ico
id: {{REDACTED}}
kind: compute#healthCheck
name: k8s1-4622dadc-{{REDACTED}}
selfLink: https://www.googleapis.com/compute/v1/projects/{{REDACTED}}
timeoutSec: 15
type: HTTP
unhealthyThreshold: 2
</code></pre>
<p>Either there's something wrong with my setup, or the BackendConfig is not equally applied for all services in the rules section of the Ingress.</p>
<p>Edit #3:</p>
<p>The Healt Check log entry for <code>port 4000</code> shows:</p>
<pre class="lang-json prettyprint-override"><code>healthCheckProbeResult: {
detailedHealthState: &quot;TIMEOUT&quot;
healthCheckProtocol: &quot;HTTP&quot;
healthState: &quot;UNHEALTHY&quot;
ipAddress: &quot;10.40.129.203&quot;
previousDetailedHealthState: &quot;UNKNOWN&quot;
previousHealthState: &quot;UNHEALTHY&quot;
probeCompletionTimestamp: &quot;2022-02-07T20:06:10.955412154Z&quot;
probeRequest: &quot;/favicon.ico&quot;
probeResultText: &quot;HTTP response: , Error: Connection refused&quot;
probeSourceIp: &quot;35.191.12.114&quot;
responseLatency: &quot;0.000569s&quot;
targetIp: &quot;10.40.129.203&quot;
targetPort: 5000
}
</code></pre>
<p>EDIT #4:
I needed to adjust the BackendConfig annotation, as my case was in fact more involved compared to what my previous definitions were showing.</p>

m0r0n
<p>The problem was in the way I applied the BackendConfig annotation of the service.
My assumption was that the <code>&quot;default&quot;</code> config would be applied to both ports and the extra header config would <em>additionally</em> be applied to port 4001.</p>
<p>But this is not the case.</p>
<p>In case a BackendConfig deviates from the default, you'll have to define a one for each case where it deviates from the HTTP ports.
BackendConfigs will not get merged and you can not add multiple configs per port.</p>
<pre class="lang-yaml prettyprint-override"><code>---
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: config-4000
spec:
  healthCheck:
    type: HTTP
    port: 5000
    requestPath: &quot;/favicon.ico&quot;
    checkIntervalSec: 20
---
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: config-4001
spec:
  healthCheck:
    type: HTTP
    port: 5000
    requestPath: &quot;/favicon.ico&quot;
    checkIntervalSec: 20
  headers:
    - &quot;X-Client-Region:{client_region}&quot;
---
apiVersion: v1
kind: Service
metadata:
  name: some-app-service
  annotations:
    cloud.google.com/backend-config: '{&quot;ports&quot;:{
       &quot;4000&quot;:&quot;config-4000&quot;,
       &quot;4001&quot;:&quot;config-4001&quot;
    }}'
spec:
  type: NodePort
  selector:
    app: some-app
  ports:
    - name: web
      targetPort: 5000
      protocol: TCP
      port: 4000
    - name: admin
      targetPort: 5001
      protocol: TCP
      port: 4001
</code></pre>
<p>EDIT #1:</p>
<p>An additional issue (resulting in the Connection Error) was in my project config:</p>
<ul>
<li>The server was configured to bind to <code>127.0.0.1</code> instead of <code>0.0.0.0</code></li>
</ul>
