<p>I am getting a <code>connection refused</code> error when trying to connect to a k8s workload.</p>
<p>This is a custom installation of k8s using kubeadm.</p>
<p>Domain: <code>example.com</code> resolves to the IP address of the k8s server.</p>
<p>I have deployed a sample pod (nginx) with an accompanying service. I can see that it works by navigating to the cluster ip (the internal 10.0.0.0/24 range has been routed to my k8s master node so it is accessible directly).</p>
<p>On the master node I can't see anything listening to port 80 using netstat.
I can also run a custom web server on port 80 (using python3 -m http.server 80) which succeeds in serving the local directory which means that no other process does.</p>
<p>Ask me anything. I will provide logs.</p>
<h3>Kubernetes</h3>
<ul>
<li>v.1.27.3</li>
</ul>
<h3>Istio</h3>
<ul>
<li>client version: 1.17.1</li>
<li>control plane version: 1.18.0</li>
<li>data plane version: 1.18.0 (1 proxies)</li>
</ul>
<h2>gateway.yaml</h2>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: gw-foo
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - example.com
    port:
      name: http
      number: 80
      protocol: HTTP
</code></pre>
<h2>virtualservice.yaml</h2>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: plex-vs
  namespace: default
spec:
  gateways:
  - default/gw-foo
  hosts:
  - example.com
  http:
  - match:
    - port: 80
    name: foo
    route:
    - destination:
        host: foo-svc.some-namespace.svc.cluster.local
        port:
          number: 30000
</code></pre>
<h3>Custom kubeadm-config.yaml</h3>
<pre class="lang-yaml prettyprint-override"><code># kubeadm-config.yaml
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
kubernetesVersion: v1.27.3
networking:
  podSubnet: &quot;10.3.0.0/24&quot;
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
#serverTLSBootstrap: true
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: &quot;192.168.1.200&quot;
</code></pre>
<h3>Istio Ingress status</h3>
<pre class="lang-bash prettyprint-override"><code>$ kubectl -n istio-ingress get deployment istio-ingressgateway 
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
istio-ingressgateway   1/1     1            1           47h
</code></pre>
<pre class="lang-bash prettyprint-override"><code>$ kubectl -n istio-ingress describe deployments.apps istio-ingressgateway 
Name:                   istio-ingressgateway
Namespace:              istio-ingress
CreationTimestamp:      Tue, 11 Jul 2023 21:55:48 +0300
Labels:                 app=istio-ingressgateway
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=istio-ingressgateway
                        app.kubernetes.io/version=1.18.0
                        helm.sh/chart=gateway-1.18.0
                        istio=ingressgateway
Annotations:            deployment.kubernetes.io/revision: 1
                        meta.helm.sh/release-name: istio-ingressgateway
                        meta.helm.sh/release-namespace: istio-ingress
Selector:               app=istio-ingressgateway,istio=ingressgateway
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=istio-ingressgateway
                    istio=ingressgateway
                    sidecar.istio.io/inject=true
  Annotations:      inject.istio.io/templates: gateway
                    prometheus.io/path: /stats/prometheus
                    prometheus.io/port: 15020
                    prometheus.io/scrape: true
                    sidecar.istio.io/inject: true
  Service Account:  istio-ingressgateway
  Containers:
   istio-proxy:
    Image:      auto
    Port:       15090/TCP
    Host Port:  0/TCP
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:        100m
      memory:     128Mi
    Environment:  &lt;none&gt;
    Mounts:       &lt;none&gt;
  Volumes:        &lt;none&gt;
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  &lt;none&gt;
NewReplicaSet:   istio-ingressgateway-778d69499b (1/1 replicas created)
Events:          &lt;none&gt;
</code></pre>
<h3>Istioctl analyze output</h3>
<pre class="lang-bash prettyprint-override"><code>$ istioctl analyze -A
Warning [IST0108] (Pod istio-ingress/istio-ingressgateway-778d69499b-g6mkv) Unknown annotation: istio.io/rev
Warning [IST0108] (Pod istio-system/istiod-5f859db56c-796zw) Unknown annotation: ambient.istio.io/redirection
Warning [IST0108] (Pod default/my-nginx-7754db7798-ft46j) Unknown annotation: istio.io/rev
Info [IST0102] (Namespace calico-apiserver) The namespace is not enabled for Istio injection. Run 'kubectl label namespace calico-apiserver istio-injection=enabled' to enable it, or 'kubectl label namespace calico-apiserver istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0102] (Namespace calico-system) The namespace is not enabled for Istio injection. Run 'kubectl label namespace calico-system istio-injection=enabled' to enable it, or 'kubectl label namespace calico-system istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0102] (Namespace cert-manager) The namespace is not enabled for Istio injection. Run 'kubectl label namespace cert-manager istio-injection=enabled' to enable it, or 'kubectl label namespace cert-manager istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0102] (Namespace hello-kubernetes) The namespace is not enabled for Istio injection. Run 'kubectl label namespace hello-kubernetes istio-injection=enabled' to enable it, or 'kubectl label namespace hello-kubernetes istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0102] (Namespace istio-ingress) The namespace is not enabled for Istio injection. Run 'kubectl label namespace istio-ingress istio-injection=enabled' to enable it, or 'kubectl label namespace istio-ingress istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0102] (Namespace tigera-operator) The namespace is not enabled for Istio injection. Run 'kubectl label namespace tigera-operator istio-injection=enabled' to enable it, or 'kubectl label namespace tigera-operator istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0118] (Service calico-apiserver/calico-api) Port name apiserver (port: 443, targetPort: 5443) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service calico-system/calico-kube-controllers-metrics) Port name metrics-port (port: 9094, targetPort: 9094) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service calico-system/calico-typha) Port name calico-typha (port: 5473, targetPort: calico-typha) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service hello-kubernetes/hello-world-service) Port name hello-svc (port: 8065, targetPort: 31870) doesn't follow the naming convention of Istio port.
</code></pre>

sm0ke21
<p>Figured it out.</p>
<p>Here's what was going wrong with my understanding.</p>
<p>I was expecting some process to be listening on port 80 and 443 on my master host.</p>
<p>That is simply not the case as demonstrated below:</p>
<pre class="lang-bash prettyprint-override"><code>kubectl -n istio-system get svc istio-ingressgateway
NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                      AGE
istio-ingressgateway   LoadBalancer   10.107.128.144   &lt;pending&gt;     15021:31095/TCP,80:32657/TCP,443:30544/TCP,31400:31995/TCP,15443:30016/TCP   58m
</code></pre>
<p>All I had to do was to route all unsecure http traffic to port 32657 and secure https traffic to port 30544 on the master host.</p>
