<p>To address the default AWS VPC CNI max pod number node limit (Max Pods = Maximum supported Network Interfaces for instance type) * ( IPv4 Addresses per Interface) - i.e. 17 in a t3.medium) in my EKS cluster I started using Weave CNI Plugin to overcome that limitation.</p>

<p>This brings a caveat, if you have an application or container running in the overlay network and the Kubernetes master node / API needs to talk to it, it won't work. For instance, the ApiService <code>v1beta1.metrics.k8s.io</code> tries to connect to <code>metrics-server</code> pods running in the overlay network and it won't succeed.
A proposed solution is to run <code>metrics-server</code> with <code>hostNetwork: true</code> which works just fine.</p>

<p>My problems started when we decided to have <code>custom-metrics</code> with <code>prometheus-adapter</code> so that we could have metrics from Kafka consumer group lag for horizontal autoscaling of the consumer pods.
To have the ApiService <code>v1beta1.custom.metrics.k8s.io</code> talking with <code>prometheus-adapter</code> pod we also have to set <code>hostNetwork: true</code> but this time the <code>prometheus-adapter</code> cannot access the Prometheus running in the overlay network anymore - and we cannot move everything to the host network!</p>

<p>I'm kind of in a dead-end here. I guess I could use some "tool" to forward the metrics that I need from the overlay network Prometheus to another Prometheus in the host network which would be then used by prometheus-adapter?</p>

<p>Problem solved. I was testing connectivity with telnet but somehow wget works just fine.</p>
