<p>Gist: I am struggling to get a pod to connect to a service outside the cluster.
Basically the pod manages to resolve the ClusterIp of the selectorless service, but traffic does not go through. Traffic does go through if i hit the ClusterIp of the selectorless service from the cluster host.</p>
<p>I'm fairly new with microk8s and k8s in general. I hope i am making some sense though...</p>
<p>Background:</p>
<p>I am attempting to move parts of my infrastructure from a docker-compose setup on one virtual machine, to a microk8s cluster (with 2 nodes).</p>
<p>In the docker compose, i have a Grafana Container, connecting to an InfluxDb container.</p>
<p>kubectl version:</p>
<pre><code>Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;22+&quot;, GitVersion:&quot;v1.22.2-3+9ad9ee77396805&quot;, GitCommit:&quot;9ad9ee77396805781cd0ae076d638b9da93477fd&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-09-30T09:52:57Z&quot;, GoVersion:&quot;go1.16.8&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
</code></pre>
<p>I now want to setup a Grafana container on the microk8s cluster, and have it connect to the InfluxDb that is still running on the docker-compose vm.</p>
<p>All of these VM's are running on an ESXi host.</p>
<ul>
<li>InfluxDb is exposed at 10.1.2.220:8086</li>
<li>microk8s-master has ip 10.1.2.50</li>
<li>microk8s-slave-1 has ip 10.1.2.51</li>
</ul>
<p>I have enabled ingress and dns. I have also enabled metallb, though i don't intend to use it here.</p>
<p>I have configured a selectorless service, a remote endpoint and an egress Network Policy (currently allowing all).</p>
<p>From microk8s-master and slave-1, i can</p>
<ul>
<li>telnet directly to 10.1.2.220:8086 successfully</li>
<li><strong>telnet to the ClusterIP(10.152.183.26):8086 of the service, successfully reaching influxdb</strong></li>
<li>wget ClusterIp:8086</li>
</ul>
<p>Inside the Pod, if i do a wget to influxdb-service:8086, it will resolve to the ClusterIP, but after that it times out.
I can however reach (wget), services pointing to other pods in the same namespace</p>
<p><strong>Update:</strong></p>
<p>I have been able to get it to work through a workaround, but i dont think this is the correct way.</p>
<p>My temporary solution is to expose the selectorless service on metallb, then use that exposed ip inside the pod.</p>
<p>Service and Endpoints for InfluxDb</p>
<pre><code>---
apiVersion: v1
kind: Service
metadata:
  name: influxdb-service
  labels:
    app: grafana
spec:
  ports:
    - protocol: TCP
      port: 8086
      targetPort: 8086
---
apiVersion: v1
kind: Endpoints
metadata:
  name: influxdb-service
subsets:
  - addresses:
      - ip: 10.1.2.220
    ports:
      - port: 8086
</code></pre>
<p>The service and endpoint shows up fine</p>
<pre><code>eso@microk8s-master:~/k8s-grafana$ microk8s.kubectl get endpoints
NAME                ENDPOINTS                             AGE
neo4j-service-lb    10.1.166.176:7687,10.1.166.176:7474   25h
influxdb-service    10.1.2.220:8086                       127m
questrest-service   10.1.166.178:80                       5d
kubernetes          10.1.2.50:16443,10.1.2.51:16443       26d
grafana-service     10.1.237.120:3000                     3h11m

eso@microk8s-master:~/k8s-grafana$ microk8s.kubectl get svc
NAME                TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
kubernetes          ClusterIP      10.152.183.1     &lt;none&gt;        443/TCP                         26d
questrest-service   ClusterIP      10.152.183.56    &lt;none&gt;        80/TCP                          5d
neo4j-service-lb    LoadBalancer   10.152.183.166   10.1.2.60     7474:31974/TCP,7687:32688/TCP   25h
grafana-service     ClusterIP      10.152.183.75    &lt;none&gt;        3000/TCP                        3h13m
influxdb-service    ClusterIP      10.152.183.26    &lt;none&gt;        8086/TCP                        129m

eso@microk8s-master:~/k8s-grafana$ microk8s.kubectl get networkpolicy
NAME                            POD-SELECTOR    AGE
grafana-allow-egress-influxdb   app=grafana     129m
test-egress-influxdb            app=questrest   128m
</code></pre>
<p>Describe:</p>
<pre><code>eso@microk8s-master:~/k8s-grafana$ microk8s.kubectl describe svc influxdb-service
Name:              influxdb-service
Namespace:         default
Labels:            app=grafana
Annotations:       &lt;none&gt;
Selector:          &lt;none&gt;
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.152.183.26
IPs:               10.152.183.26
Port:              &lt;unset&gt;  8086/TCP
TargetPort:        8086/TCP
Endpoints:         10.1.2.220:8086
Session Affinity:  None
Events:            &lt;none&gt;

eso@microk8s-master:~/k8s-grafana$ microk8s.kubectl describe endpoints influxdb-service
Name:         influxdb-service
Namespace:    default
Labels:       &lt;none&gt;
Annotations:  &lt;none&gt;
Subsets:
  Addresses:          10.1.2.220
  NotReadyAddresses:  &lt;none&gt;
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    &lt;unset&gt;  8086  TCP

Events:  &lt;none&gt;

eso@microk8s-master:~/k8s-grafana$ microk8s.kubectl describe networkpolicy grafana-allow-egress-influxdb
Name:         grafana-allow-egress-influxdb
Namespace:    default
Created on:   2021-11-03 20:53:00 +0000 UTC
Labels:       &lt;none&gt;
Annotations:  &lt;none&gt;
Spec:
  PodSelector:     app=grafana
  Not affecting ingress traffic
  Allowing egress traffic:
    To Port: &lt;any&gt; (traffic allowed to all ports)
    To: &lt;any&gt; (traffic not restricted by destination)
  Policy Types: Egress

</code></pre>
<p>Grafana.yml:</p>
<pre><code>eso@microk8s-master:~/k8s-grafana$ cat grafana.yml
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  storageClassName: &quot;&quot;
  claimRef:
    name: grafana-pvc
    namespace: default
  persistentVolumeReclaimPolicy: Retain
  nfs:
    path: /mnt/MainVol/grafana
    server: 10.2.0.1
    readOnly: false
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: &quot;&quot;
  volumeName: grafana-pv
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
spec:
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        fsGroup: 472
        supplementalGroups:
          - 0
      containers:
        - name: grafana
          image: grafana/grafana:7.5.2
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http-grafana
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /robots.txt
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 2
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3000
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 250m
              memory: 750Mi
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-pv
      volumes:
        - name: grafana-pv
          persistentVolumeClaim:
            claimName: grafana-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: grafana-service
spec:
  ports:
    - port: 3000
      protocol: TCP
      targetPort: http-grafana
  selector:
    app: grafana
  #sessionAffinity: None
  #type: LoadBalancer
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: &quot;g2.some.domain.com&quot;
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grafana-service
            port:
              number: 3000
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: grafana-allow-egress-influxdb
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: grafana
  ingress:
  - {}
  egress:
  - {}
  policyTypes:
  - Egress

</code></pre>

takilara
<p>As I haven't gotten much response, i'll answer the question with my &quot;workaround&quot;. I am still not sure this is the best way to do it though.</p>
<p>I got it to work by exposing the selectorless service on metallb, then using that exposed ip inside grafana</p>
<pre><code>kind: Service
apiVersion: v1
metadata:
  name: influxdb-service-lb
  #namespace: ingress
spec:
  type: LoadBalancer
  loadBalancerIP: 10.1.2.61
#  selector:
#    app: grafana
  ports:
  - name: http
    protocol: TCP
    port: 8086
    targetPort: 8086
---
apiVersion: v1
kind: Endpoints
metadata:
  name: influxdb-service-lb
subsets:
  - addresses:
      - ip: 10.1.2.220
    ports:
      - name: influx
        protocol: TCP
        port: 8086
</code></pre>
<p>I then use the loadbalancer ip in grafana (10.1.2.61)</p>
<hr />
<p>Update October 2022
As a response to a comment above, I have added a diagram of how i believe this to work</p>
<p><a href="https://i.stack.imgur.com/3icoz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/3icoz.png" alt="enter image description here" /></a></p>
