<p>I have too many LoadBalancer services consuming too many external IPs and I'd like to switch to using an Ingress controller.</p>

<p>I did the <a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer" rel="nofollow noreferrer">tutorial</a> and everything worked fine with the google provided pods.</p>

<p>However, with my pod I am able to hit the NodePort service ...</p>

<pre><code> 😈   &gt;curl http://35.223.89.81:32607/healthz
OK 😈   &gt;
</code></pre>

<p>... but calls to the Ingress Controller are consistently failing ...</p>

<pre><code> 😈   &gt;curl http://35.241.21.71:80/healthz
&lt;!DOCTYPE html&gt;
&lt;html lang=en&gt;
  &lt;meta charset=utf-8&gt;
  &lt;meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width"&gt;
  &lt;title&gt;Error 404 (Not Found)!!1&lt;/title&gt;
  &lt;style&gt;
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* &gt; body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  &lt;/style&gt;
  &lt;a href=//www.google.com/&gt;&lt;span id=logo aria-label=Google&gt;&lt;/span&gt;&lt;/a&gt;
  &lt;p&gt;&lt;b&gt;404.&lt;/b&gt; &lt;ins&gt;That’s an error.&lt;/ins&gt;
  &lt;p&gt;The requested URL &lt;code&gt;/healthz&lt;/code&gt; was not found on this server.  &lt;ins&gt;That’s all we know.&lt;/ins&gt;
</code></pre>

<p>This is the version of k8s I am using:</p>

<pre><code> 😈   &gt;gcloud container clusters list
NAME              LOCATION       MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION     NUM_NODES  STATUS
monza-predictors  us-central1-a  1.13.11-gke.14  35.193.247.210  n1-standard-1  1.13.11-gke.9 *  2          RUNNING
</code></pre>

<p><strong>YAML for the ingress</strong></p>

<pre><code> 😈   &gt;cat fanout-ingress-v2.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: fanout-ingress
spec:
  rules:
  - http:
      paths:
      - path: /healthz
        backend:
          serviceName: predictor-classification-seatbelt-driver-service-node-port
          servicePort: 4444
      - path: /seatbelt-driver
        backend:
          serviceName: predictor-classification-seatbelt-driver-service-node-port
          servicePort: 4444
</code></pre>

<p><strong>describe of the ingress</strong></p>

<pre><code> 😈   &gt;kubectl describe ing fanout-ingress 
Name:             fanout-ingress
Namespace:        default
Address:          35.241.21.71
Default backend:  default-http-backend:80 (10.40.2.10:8080)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     
        /healthz           predictor-classification-seatbelt-driver-service-node-port:4444 (&lt;none&gt;)
        /seatbelt-driver   predictor-classification-seatbelt-driver-service-node-port:4444 (&lt;none&gt;)
Annotations:
  ingress.kubernetes.io/url-map:                     k8s-um-default-fanout-ingress--62f4c45447b62142
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"fanout-ingress","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"predictor-classification-seatbelt-driver-service-node-port","servicePort":4444},"path":"/healthz"},{"backend":{"serviceName":"predictor-classification-seatbelt-driver-service-node-port","servicePort":4444},"path":"/seatbelt-driver"}]}}]}}

  ingress.kubernetes.io/backends:         {"k8s-be-31413--62f4c45447b62142":"HEALTHY","k8s-be-32607--62f4c45447b62142":"UNHEALTHY"}
  ingress.kubernetes.io/forwarding-rule:  k8s-fw-default-fanout-ingress--62f4c45447b62142
  ingress.kubernetes.io/target-proxy:     k8s-tp-default-fanout-ingress--62f4c45447b62142
Events:
  Type    Reason  Age   From                     Message
  ----    ------  ----  ----                     -------
  Normal  ADD     21m   loadbalancer-controller  default/fanout-ingress
  Normal  CREATE  19m   loadbalancer-controller  ip: 35.241.21.71
</code></pre>

<p>I noticed that 1 of the 2 backends are UNHEALTHY.</p>

<p><strong>YAML for the NodePort service:</strong></p>

<pre><code> 😈   &gt;cat service-node-port-classification-predictor.yaml
apiVersion: v1
kind: Service
metadata:
  name: predictor-classification-seatbelt-driver-service-node-port
  namespace: default
spec:
  ports:
  - port: 4444
    protocol: TCP
    targetPort: 4444
  selector:
    app: predictor-classification-seatbelt-driver
  type: NodePort
</code></pre>

<p><strong>describe of the NodePort service</strong></p>

<pre><code> 😈   &gt;kubectl describe svc predictor-classification-seatbelt-driver-service-node-port
Name:                     predictor-classification-seatbelt-driver-service-node-port
Namespace:                default
Labels:                   &lt;none&gt;
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"predictor-classification-seatbelt-driver-service-node-port","name...
Selector:                 app=predictor-classification-seatbelt-driver
Type:                     NodePort
IP:                       10.43.243.69
Port:                     &lt;unset&gt;  4444/TCP
TargetPort:               4444/TCP
NodePort:                 &lt;unset&gt;  32607/TCP
Endpoints:                10.40.2.16:4444
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   &lt;none&gt;
</code></pre>

<p><strong>YAML for the deployment</strong></p>

<pre><code> 😈   &gt;cat deployment-classification-predictor-v2.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: predictor-classification-seatbelt-driver
  labels:
    app: predictor-classification-seatbelt-driver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: predictor-classification-seatbelt-driver
  template:
    metadata:
      labels:
        app: predictor-classification-seatbelt-driver
    spec:
      containers:
      - name: predictor-classification-seatbelt-driver
        image: gcr.io/annotator-1286/classification-predictor
        command: ["/app/server.sh"]
        args: ["4444", "https://storage.googleapis.com/com-aosvapps-runs/38/1564677191/models/mobile.pb", "https://storage.googleapis.com/com-aosvapps-runs/38/1564677191/models/labels.csv"]
        ports:
        - containerPort: 4444
        livenessProbe:
          httpGet:
            path: /healthz
            port: 4444
          initialDelaySeconds: 120
</code></pre>

<p><strong>describe of the deployment</strong></p>

<pre><code> 😈   &gt;kubectl describe deploy predictor-classification-seatbelt-driver
Name:                   predictor-classification-seatbelt-driver
Namespace:              default
CreationTimestamp:      Mon, 18 Nov 2019 12:17:13 -0800
Labels:                 app=predictor-classification-seatbelt-driver
Annotations:            deployment.kubernetes.io/revision: 1
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"predictor-classification-seatbelt-driver"},"name...
Selector:               app=predictor-classification-seatbelt-driver
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=predictor-classification-seatbelt-driver
  Containers:
   predictor-classification-seatbelt-driver:
    Image:      gcr.io/annotator-1286/classification-predictor
    Port:       4444/TCP
    Host Port:  0/TCP
    Command:
      /app/server.sh
    Args:
      4444
      https://storage.googleapis.com/com-aosvapps-runs/38/1564677191/models/mobile.pb
      https://storage.googleapis.com/com-aosvapps-runs/38/1564677191/models/labels.csv
    Liveness:     http-get http://:4444/healthz delay=120s timeout=1s period=10s #success=1 #failure=3
    Environment:  &lt;none&gt;
    Mounts:       &lt;none&gt;
  Volumes:        &lt;none&gt;
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  &lt;none&gt;
NewReplicaSet:   predictor-classification-seatbelt-driver-85bc679444 (1/1 replicas created)
Events:          &lt;none&gt;
</code></pre>

<p><strong>describe of the pod</strong></p>

<pre><code> 😈   &gt;kubectl describe po predictor-classification-seatbelt-driver-85bc679444-lcb7v
Name:               predictor-classification-seatbelt-driver-85bc679444-lcb7v
Namespace:          default
Priority:           0
PriorityClassName:  &lt;none&gt;
Node:               gke-monza-predictors-default-pool-268f57e3-1bs6/10.128.0.65
Start Time:         Mon, 18 Nov 2019 12:17:13 -0800
Labels:             app=predictor-classification-seatbelt-driver
                    pod-template-hash=85bc679444
Annotations:        kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container predictor-classification-seatbelt-driver
Status:             Running
IP:                 10.40.2.16
Controlled By:      ReplicaSet/predictor-classification-seatbelt-driver-85bc679444
Containers:
  predictor-classification-seatbelt-driver:
    Container ID:  docker://90ce1466b852760db92bc66698295a2ae2963f19d26111e5be03d588dc83a712
    Image:         gcr.io/annotator-1286/classification-predictor
    Image ID:      docker-pullable://gcr.io/annotator-1286/classification-predictor@sha256:63690593d710182110e51fbd620d6944241c36dd79bce7b08b2823677ec7b929
    Port:          4444/TCP
    Host Port:     0/TCP
    Command:
      /app/server.sh
    Args:
      4444
      https://storage.googleapis.com/com-aosvapps-runs/38/1564677191/models/mobile.pb
      https://storage.googleapis.com/com-aosvapps-runs/38/1564677191/models/labels.csv
    State:          Running
      Started:      Mon, 18 Nov 2019 12:17:15 -0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
    Liveness:     http-get http://:4444/healthz delay=120s timeout=1s period=10s #success=1 #failure=3
    Environment:  &lt;none&gt;
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-8q95m (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-8q95m:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-8q95m
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          &lt;none&gt;
</code></pre>

<p><strong>UPDATE: Using a <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/" rel="nofollow noreferrer">Single Service Ingress</a> did not fix the problem</strong></p>

<pre><code> 😈   &gt;cat fanout-ingress-v3.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: fanout-ingress
spec:
  backend:
    serviceName: predictor-classification-seatbelt-driver-service-node-port
    servicePort: 4444
 😈   &gt;kubectl apply -f fanout-ingress-v3.yaml 
ingress.extensions/fanout-ingress created


 😈   &gt;kubectl describe ing fanout-ingress 
Name:             fanout-ingress
Namespace:        default
Address:          35.244.250.224
Default backend:  predictor-classification-seatbelt-driver-service-node-port:4444 (10.40.2.16:4444)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     *     predictor-classification-seatbelt-driver-service-node-port:4444 (10.40.2.16:4444)
Annotations:
  ingress.kubernetes.io/url-map:                     k8s-um-default-fanout-ingress--62f4c45447b62142
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"fanout-ingress","namespace":"default"},"spec":{"backend":{"serviceName":"predictor-classification-seatbelt-driver-service-node-port","servicePort":4444}}}

  ingress.kubernetes.io/backends:         {"k8s-be-32607--62f4c45447b62142":"Unknown"}
  ingress.kubernetes.io/forwarding-rule:  k8s-fw-default-fanout-ingress--62f4c45447b62142
  ingress.kubernetes.io/target-proxy:     k8s-tp-default-fanout-ingress--62f4c45447b62142
Events:
  Type    Reason  Age    From                     Message
  ----    ------  ----   ----                     -------
  Normal  ADD     3m31s  loadbalancer-controller  default/fanout-ingress
  Normal  CREATE  2m56s  loadbalancer-controller  ip: 35.244.250.224



😈   &gt;curl 35.244.250.224/healthz
&lt;!DOCTYPE html&gt;
&lt;html lang=en&gt;
  &lt;meta charset=utf-8&gt;
  &lt;meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width"&gt;
  &lt;title&gt;Error 404 (Not Found)!!1&lt;/title&gt;
  &lt;style&gt;
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* &gt; body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  &lt;/style&gt;
  &lt;a href=//www.google.com/&gt;&lt;span id=logo aria-label=Google&gt;&lt;/span&gt;&lt;/a&gt;
  &lt;p&gt;&lt;b&gt;404.&lt;/b&gt; &lt;ins&gt;That’s an error.&lt;/ins&gt;
  &lt;p&gt;The requested URL &lt;code&gt;/healthz&lt;/code&gt; was not found on this server.  &lt;ins&gt;That’s all we know.&lt;/ins&gt;
</code></pre>

Alex Ryan
<p>Add a <strong>readinessProbe</strong> to your <code>Deployment</code> object.</p>

<pre><code>    readinessProbe:
      httpGet:
        path: /healthz
        port: 4444
      initialDelaySeconds: 120
</code></pre>

<p>An IngressController may wait with routing traffic to the service until the pods in the service become <em>ready</em> to handle requests from the Ingress proxy.</p>
