<p>I recently upgraded my ingress and external-dns due to the Kubernetes 1.22 upgrade. I made changes where you <a href="https://github.com/rivernews/iriversland2-kubernetes/pull/54/files#diff-6c23af14bc7d46344b07eb12a534dcda49210e711d4d721a87a77cb0bbd877f9" rel="nofollow noreferrer">can see it here</a>.</p>
<p>Now the nginx controller is running on k8s w/o error, but I cannot cURL from my laptop to access the services.</p>
<p>What I've tried:</p>
<ul>
<li><code>nslookup grafana.shaungc.com</code> or <code>nslookup code-server.shaungc.com</code> gives the result. This means the external-dns is working. I'm using Route53. I do noticed all microservice endpoint points to the same public IP address <code>10.245.215.152</code>, but I guess that's normal because I'm using <code>clusterIP</code>.</li>
<li>Watch the ingress controller log (I skipped some part that seems not relevant)</li>
</ul>
<pre><code>(initially)
...
I1023 11:30:24.114812       6 controller.go:648] Replacing location &quot;/&quot; for server &quot;code-server.shaungc.com&quot; with upstream &quot;upstream-default-backend&quot; to use upstream &quot;code-server-code-server-service-8003&quot; (Ingress &quot;code-server/code-server-ingress-resource&quot;)
I1023 11:30:24.114828       6 controller.go:648] Replacing location &quot;/&quot; for server &quot;*.shaungc.com&quot; with upstream &quot;upstream-default-backend&quot; to use upstream &quot;cert-manager-dummy-svc-dummy-port&quot; (Ingress &quot;cert-manager/tls-wildcard-cert-ingress-resource&quot;)
I1023 11:30:24.114841       6 controller.go:648] Replacing location &quot;/&quot; for server &quot;*.api.shaungc.com&quot; with upstream &quot;upstream-default-backend&quot; to use upstream &quot;cert-manager-dummy-svc-dummy-port&quot; (Ingress &quot;cert-manager/tls-wildcard-cert-ingress-resource&quot;)
I1023 11:30:24.114854       6 controller.go:648] Replacing location &quot;/&quot; for server &quot;*.812c211c-6cbb-79e3-420e-92502524c690.shaungc.com&quot; with upstream &quot;upstream-default-backend&quot; to use upstream &quot;cert-manager-dummy-svc-dummy-port&quot; (Ingress &quot;cert-manager/tls-wildcard-cert-ingress-resource&quot;)
I1023 11:30:24.114874       6 controller.go:303] Obtaining information about TCP stream services from ConfigMap &quot;kube-system/nginx-ingress-controller-ingress-nginx-tcp&quot;
...
I1023 11:30:24.124150       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.198254       6 main.go:101] &quot;successfully validated configuration, accepting&quot; ingress=&quot;tls-wildcard-cert-ingress-resource/cert-manager&quot;
I1023 11:30:24.203337       6 store.go:741] updating annotations information for ingress cert-manager/tls-wildcard-cert-ingress-resource
I1023 11:30:24.203620       6 main.go:187] &quot;No default affinity found&quot; ingress=&quot;tls-wildcard-cert-ingress-resource&quot;
I1023 11:30:24.204116       6 store.go:775] updating references to secrets for ingress cert-manager/tls-wildcard-cert-ingress-resource
I1023 11:30:24.204362       6 backend_ssl.go:41] &quot;Syncing Secret&quot; name=&quot;cert-manager/wilcard-tls-ing-certificate-secret&quot;
I1023 11:30:24.203428       6 event.go:282] Event(v1.ObjectReference{Kind:&quot;Ingress&quot;, Namespace:&quot;cert-manager&quot;, Name:&quot;tls-wildcard-cert-ingress-resource&quot;, UID:&quot;a1a81692-9205-4abe-95e7-c4c0a865a49b&quot;, APIVersion:&quot;networking.k8s.io/v1&quot;, ResourceVersion:&quot;110423110&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1023 11:30:24.204981       6 ssl.go:111] &quot;parsing ssl certificate extensions&quot;
I1023 11:30:24.205151       6 backend_ssl.go:145] &quot;Configuring Secret \&quot;cert-manager/wilcard-tls-ing-certificate-secret\&quot; for TLS encryption ...
...
I1023 11:30:24.208957       6 endpoints.go:129] Endpoints found for Service &quot;code-server/code-server-service&quot;: [{10.244.0.160 8003 &amp;ObjectReference{Kind:Pod,Namespace:code-server,Name:code-server-deployment-5445c4587-64bj6,UID:b28da30d-43dd-4cf4-b365-b90e8ab0668c,APIVersion:,ResourceVersion:110407594,FieldPath:,}}]
I1023 11:30:24.209142       6 controller.go:1262] Host &quot;grafana.shaungc.com&quot; is listed in the TLS section but secretName is empty. Using default certificate
I1023 11:30:24.209310       6 controller.go:1262] Host &quot;api.shaungc.com&quot; is listed in the TLS section but secretName is empty. Using default certificate
I1023 11:30:24.209447       6 controller.go:1262] Host &quot;appl-tracky.api.shaungc.com&quot; is listed in the TLS section but secretName is empty. Using default certificate
I1023 11:30:24.209899       6 controller.go:1262] Host &quot;code-server.shaungc.com&quot; is listed in the TLS section but secretName is empty. Using default certificate
...
I1023 11:30:37.447833       6 status.go:276] &quot;skipping update of Ingress (no change)&quot; namespace=&quot;appl-tracky-api&quot; ingress=&quot;appl-tracky-api-ingress-resource&quot;
I1023 11:30:37.447842       6 status.go:276] &quot;skipping update of Ingress (no change)&quot; namespace=&quot;cert-manager&quot; ingress=&quot;tls-wildcard-cert-ingress-resource&quot;
I1023 11:30:37.447850       6 status.go:276] &quot;skipping update of Ingress (no change)&quot; namespace=&quot;code-server&quot; ingress=&quot;code-server-ingress-resource&quot;
(end of log)
</code></pre>
<ul>
<li>In the K8s dashboard I see no failed pods or resources, all green.</li>
</ul>
<p>It previously worked. But with the 1.22 k8s upgrade and my PR change, I can't access any website/microservice on the cluster from either browser or cURL.</p>
<p>What else can I try to debug at this point?</p>

Shawn
<p>It turns out that the <a href="https://github.com/helm/charts/tree/master/stable/nginx-ingress" rel="nofollow noreferrer">legacy Helm chart <code>nginx-ingress</code></a> I was using, is not the same as the <a href="https://github.com/kubernetes/ingress-nginx" rel="nofollow noreferrer">kubernetes/nginx-ingress</a> chart, they are different alternatives for nginx ingress controllers - previously their development co-exist at the same time and are different solutions for nginx ingress. It's suboptimal in my opinion that the legacy Helm chart <a href="https://github.com/helm/charts/tree/master/stable/nginx-ingress" rel="nofollow noreferrer">README deprecation note</a> merely point to <code>kubernetes/nginx-ingress</code>. It's not going to be a linear upgrade, they different in many default chart behaviors. I went through their version history, adapt syntax, even made it work nginx ingress runs perfectly w/o error, but still my external traffic can not reach my cluster service, not even getting the correct cluster Public IP. It just doesn't work with my use case.</p>
<p>The solution? Seems the <a href="https://github.com/bitnami/charts/tree/main/bitnami/nginx-ingress-controller" rel="nofollow noreferrer">Bitnami nginx controller</a> is actually the successor of the legacy helm chart. While the syntax is different - but actually just controller level configs got pull up to 1st/global level, other default behavior are pretty similar to the legacy helm and therefore it worked for my setup. In fact I was able to upgrade to latest version of Bitnami chart 9.3.18. I explained a bit more my use case <a href="https://stackoverflow.com/a/74177702/9814131">here</a>. Spent my entire weekend on this!</p>
