<p>I'm flabbergasted.</p>

<p>I have a staging and production environment. Both environments have the same deployments, services, ingress, firewall rules, and both serve a <code>200</code> on <code>/</code>. </p>

<p>However, after turning on the staging environment and provisioning the same ingress, the staging service fails with  <code>Some backend services are in UNKNOWN state</code>. Production is still live.</p>

<p>Both the frontend and backend pods are ready on GKE. I've manually tested the health checks and they pass when I visit <code>/</code>.</p>

<p>I see nothing in the logs or gcp docs pointing in the right direction. What could I have possibly broken?</p>

<p><code>ingress.yaml</code>:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: fanout-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "STATIC-IP"
spec:
  backend:
    serviceName: frontend
    servicePort: 8080
  tls:
  - hosts:
    - &lt;DOMAIN&gt;
    secretName: staging-tls
  rules:
  - host: &lt;DOMAIN&gt;
    http:
      paths:
      - path: /*
        backend:
          serviceName: frontend
          servicePort: 8080
      - path: /backend/*
        backend:
          serviceName: backend
          servicePort: 8080
</code></pre>

<p><code>frontend.yaml</code>:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  labels:
    app: frontend
  name: frontend
  namespace: default
spec:
  ports:
  - nodePort: 30664
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: frontend
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  generation: 15
  labels:
    app: frontend
  name: frontend
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  minReadySeconds: 5
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - image: &lt;our-image&gt;
        name: frontend
        ports:
        - containerPort: 8080
          protocol: TCP
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 3
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 3
</code></pre>

Mike
<p>Yesterday even this guide <a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer" rel="noreferrer">https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer</a></p>

<p>didn't work. Don't know what happened but even waiting 30minutes + the ingress was reporting UNKNOWN state for backends . </p>

<p>After 24 hours, things seem to be much better. L7 http ingress works but with big delay on reporting healthy backends. </p>
