<p>I try to use ingress for loadbalancer of 2 services on Google Kubernetes engine:</p>

<p>here is ingress config for this:</p>

<pre><code>apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: basic-ingress

spec:
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: web
          servicePort: 8080
      - path: /v2/keys
        backend:
          serviceName: etcd-np
          servicePort: 2379
</code></pre>

<p>where web is some example service from google samples:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: default
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    run: web
  type: NodePort

----
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: default
spec:
  selector:
    matchLabels:
      run: web
  template:
    metadata:
      labels:
        run: web
    spec:
      containers:
      - image: gcr.io/google-samples/hello-app:1.0
        imagePullPolicy: IfNotPresent
        name: web
        ports:
        - containerPort: 8080
          protocol: TCP
</code></pre>

<p>But the second service is ETCD cluster with NodePort service:</p>

<pre><code>---
apiVersion: v1
kind: Service
metadata:
  name: etcd-np
spec:
  ports:
  - port: 2379
    targetPort: 2379
  selector:
    app: etcd
  type: NodePort
</code></pre>

<p>But only first ingress rule works properly i see in logs:</p>

<pre><code>ingress.kubernetes.io/backends: {"k8s-be-30195--ebfd7339a961462d":"UNHEALTHY","k8s-be-30553--ebfd7339a961462d":"HEALTHY","k8s-be-31529--ebfd7339a961462d":"HEALTHY"}
</code></pre>

<p>I etcd-np works properly it is not a problem of etcd , i think that the problem is that etcd server answers with 404 on GET / request and some healthcheck on ingress level does not allow to use it . </p>

<p>Thats why i have 2 questions :</p>

<p>1 ) How can I provide healthcheck urls for each backend path on ingress </p>

<p>2 ) How can I debug such issues . What I see now is</p>

<pre><code>kubectl describe ingress basic-ingress
Name:             basic-ingress
Namespace:        default
Address:          4.4.4.4
Default backend:  default-http-backend:80 (10.52.6.2:8080)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /*         web:8080 (10.52.8.10:8080)
              /v2/keys   etcd-np:2379 (10.52.0.2:2379,10.52.2.4:2379,10.52.8.4:2379)
Annotations:  ingress.kubernetes.io/backends:
                {"k8s-be-30195--ebfd7339a961462d":"UNHEALTHY","k8s-be-30553--ebfd7339a961462d":"HEALTHY","k8s-be-31529--ebfd7339a961462d":"HEALTHY"}
              ingress.kubernetes.io/forwarding-rule: k8s-fw-default-basic-ingress--ebfd7339a961462d
              ingress.kubernetes.io/target-proxy: k8s-tp-default-basic-ingress--ebfd7339a961462d
              ingress.kubernetes.io/url-map: k8s-um-default-basic-ingress--ebfd7339a961462d
Events:       &lt;none&gt;
</code></pre>

<p>But it does not provide me any info about this incident</p>

<p><strong>UP</strong></p>

<pre><code>kubectl describe svc etcd-np

Name:                     etcd-np
Namespace:                default
Labels:                   &lt;none&gt;
Annotations:              Selector:  app=etcd
Type:                     NodePort
IP:                       10.4.7.20
Port:                     &lt;unset&gt;  2379/TCP
TargetPort:               2379/TCP
NodePort:                 &lt;unset&gt;  30195/TCP
Endpoints:                10.52.0.2:2379,10.52.2.4:2379,10.52.8.4:2379
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   &lt;none&gt;
</code></pre>

Oleg
<p>According to the <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/ingress#health_checks" rel="nofollow noreferrer">doc</a>.</p>

<blockquote>
  <p>A Service exposed through an Ingress must respond to health checks
  from the load balancer. Any container that is the final destination of
  load-balanced traffic must do one of the following to indicate that it
  is healthy:</p>
  
  <ol>
  <li>Serve a response with an <code>HTTP</code> <code>200</code> status to GET requests on the <code>/</code> path.</li>
  <li>Configure an <code>HTTP</code> readiness probe. Serve a response with an <code>HTTP</code> <code>200</code> status to <code>GET</code> requests on the path specified by the readiness
  probe. The Service exposed through an Ingress must point to the same
  container port on which the readiness probe is enabled.</li>
  </ol>
  
  <p>For example, suppose a container specifies this readiness probe:</p>

<pre><code>...
readinessProbe:
  httpGet:
    path: /healthy
</code></pre>
  
  <p>Then if the handler for the container's <code>/healthy</code> path returns an
  <code>HTTP</code> <code>200</code> status, the load balancer considers the container to be
  alive and healthy.</p>
</blockquote>

<p>Now since ETCD has a health endpoint at <code>/health</code> the readiness probe will look like</p>

<pre><code>...
readinessProbe:
  httpGet:
    path: /health
</code></pre>

<p>This becomes a bit tricky if mTLS is enabled in ETCD. To avoid that check the <a href="https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/monitoring.md#health-check" rel="nofollow noreferrer">docs</a>.</p>
