<p>Spark version :2.4.4</p>
<p>k8s version : 1.18</p>
<p>I have a Spark and a k8s cluster.</p>
<p>I followed Spark documentation : <a href="https://spark.apache.org/docs/2.4.4/running-on-kubernetes.html" rel="nofollow noreferrer">https://spark.apache.org/docs/2.4.4/running-on-kubernetes.html</a></p>
<p>When I submit a job with an HTTP proxy on k8s : everything is ok.</p>
<p>However with the native HTTPS API on k8s I got this error :</p>
<p>Previously I had to import k8s API cert to my master Spark (keytool).</p>
<pre><code>internal.WatchConnectionManager: Exec Failure: HTTP 403, Status: 403 - pods &quot;spark-pi-1598541432880-driver&quot; is forbidden: User &quot;system:anonymous&quot; cannot watch resource &quot;pods&quot; in API group &quot;&quot; in the namespace &quot;default&quot;
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
    at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:216)
    at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:183)
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

</code></pre>
<p>This is curious because there is no user anonymous.</p>
<p>I already try to add an 'anonymous' user to k8s but it has no effect.</p>
<p>If I try without import the API cert I got this error :</p>
<pre><code>Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:198)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1967)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:331)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:325)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1688)

</code></pre>
<p>My command to submit my job :</p>
<pre><code>-bash-4.2$ spark-submit --master k8s://https://ip:port  --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=3 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.container.image=docker.io/spawnxx/spark:fink-test-2 hdfs://hdfs_ip/user/sacha.pateyron/spark-examples_2.11-2.4.4.jar 1000
</code></pre>
<p>K8s https API is not supported natively by Apache Spark (so I must import the cert) ?</p>
<p>Any idea ?</p>

<p>Solution :</p>
<p>HTTPS k8s API use cert and token for authentication.</p>
<p>First download k8s HTTPS API :</p>
<p>On master spark -&gt;</p>
<pre><code>echo -n|openssl s_client -connect ip_master_k8s:port_https_api|openssl x509 -outform PEM &gt; selfsigned_certificate.pem
</code></pre>
<p>On k8s master machine get spark token :</p>
<pre><code>kubectl get secret

kubectl describe secret spark-token-XXX
</code></pre>
<p>Then on Spark master we submit a job with cert and token :</p>
<pre><code>spark-submit --master k8s://https://ip_master_k8s:port_https_api  --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=3 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.container.image=your_image --conf spark.kubernetes.authenticate.submission.caCertFile=selfsigned_certificate.pem --conf spark.kubernetes.authenticate.submission.oauthToken=spark-token-XXX hdfs://ip_master_hdfs/my_jar
</code></pre>
