<p>I'm using <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator" rel="nofollow noreferrer">google spark-operator</a> and some <a href="https://docs.projectcalico.org/reference/resources/networkpolicy#rule" rel="nofollow noreferrer">calico</a> network policies to protect the namespaces.</p>
<p>The Spark driver pods need to be able to communicate with the <code>kubernetes</code> service in the <code>default</code> namespace to speak with the api-server.<br />
This is what I get :</p>
<pre><code>Operation: [get]  for kind: [Pod]  with name: [xx]  in namespace: [xx]  failed.
</code></pre>
<p><em><strong>The problem is :</strong><br />
Using any kind of network policies blocks communication toward the default namespace. Restoring the connectivity is possible but selecting the <code>kubernetes.default</code> service is still impossible as it is a particular service (has no selectors)... And so you can't communicate with it !</em></p>
<hr />
<p>I tried opening communication to all pods in <code>default</code> + <code>kube-system</code> namespace. It's working for all services <strong>except</strong> <code>kubernetes.default</code> which is still unreachable !</p>
<pre><code>apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: mynetpol
spec:
  selector: all()
  types:
    - Egress
  egress:

    # Allows comm to kube-system namespace
    - action: Allow
      destination:
        selector: all()
        namespaceSelector: ns == 'kube-system'
    - action: Allow
      source:
        selector: all()
        namespaceSelector: ns == 'kube-system'

   # Allows comm to default namespace
    - action: Allow
      destination:
        selector: all()
        namespaceSelector: ns == 'default'
    - action: Allow
      source:
        selector: all()
        namespaceSelector: ns == 'default'
</code></pre>
<p>For some reasons curling <code>kubernetes.default.svc.cluster.local:443</code> timeouts event though all communication is wide open.</p>

Doctor
<p>So... In the end...</p>
<p>Network policies don't work on services that dont target pods, which is the case of this particular <code>kubernetes</code> service sitting quietly in the <code>default</code> namespace. It's a special service that always points to the api-server.</p>
<hr />
<p>The solution is to retrieve the api-server's real IP and allow <code>egress</code>-ing to it.</p>
<p>To find this IP you can use this command :</p>
<pre><code>kubectl get endpoints --namespace default kubernetes
</code></pre>
<p><em>Courtesy of @Dave McNeill</em></p>
<p>Then you can allow this IP in you network policy.</p>
<ul>
<li><p>If you are using the default netpol API, check Dave's answer <a href="https://stackoverflow.com/a/56494510/5512455">https://stackoverflow.com/a/56494510/5512455</a></p>
</li>
<li><p>If you are using the calico policies, which I encourage you to do because the Kube ones sucks, following is the working yaml:</p>
</li>
</ul>
<pre><code>kind: NetworkPolicy
metadata:
  name: allow-egress-api-server
spec:
  selector: all()
  types:
    - Egress
  egress:

    # Allow api-server
    - action: Allow
      protocol: TCP
      destination:
        nets:
        - &lt;Your api-server IP&gt;/32
        ports:
        - 6443
</code></pre>
