<p>I have a simple Flink streaming app. It runs well in a cluster created by <code>start-cluster.sh</code> command.</p>
<p>Now based on the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#application-mode" rel="nofollow noreferrer">Flink tutorial</a>, I hope to deploy it in application mode natively in a Kubernetes cluster created by <a href="https://k3d.io/" rel="nofollow noreferrer">k3d</a> on macOS.</p>
<p>First, I created a cluster by <code>k3d cluster create dev</code>.</p>
<p>Here is my <strong>Dockerfile</strong>:</p>
<pre><code>FROM flink
RUN mkdir -p $FLINK_HOME/usrlib
COPY target/streaming-0.1.jar $FLINK_HOME/usrlib/streaming-0.1.jar
</code></pre>
<p>I built and pushed it to Docker Hub.</p>
<p>My cluster name is <code>k3d-dev</code>, so I ran</p>
<pre><code>flink run-application \
    --target kubernetes-application \
    -Dkubernetes.cluster-id=k3d-dev \
    -Dkubernetes.container.image=hongbomiao/my-flink-xxx:latest \
    local:///opt/flink/usrlib/streaming-0.1.jar

</code></pre>
<p>However, I got error:</p>
<pre><code> The program finished with the following exception:

io.fabric8.kubernetes.client.KubernetesClientException: JcaPEMKeyConverter is provided by BouncyCastle, an optional dependency. To use support for EC Keys you must explicitly add this dependency to classpath.
    at io.fabric8.kubernetes.client.internal.CertUtils.handleECKey(CertUtils.java:161)
    at io.fabric8.kubernetes.client.internal.CertUtils.loadKey(CertUtils.java:131)
    at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:111)
    at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:243)
    at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128)
    at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:122)
    at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:82)
    at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:62)
    at io.fabric8.kubernetes.client.BaseClient.&lt;init&gt;(BaseClient.java:51)
    at io.fabric8.kubernetes.client.DefaultKubernetesClient.&lt;init&gt;(DefaultKubernetesClient.java:105)
    at org.apache.flink.kubernetes.kubeclient.FlinkKubeClientFactory.fromConfiguration(FlinkKubeClientFactory.java:102)
    at org.apache.flink.kubernetes.KubernetesClusterClientFactory.createClusterDescriptor(KubernetesClusterClientFactory.java:61)
    at org.apache.flink.kubernetes.KubernetesClusterClientFactory.createClusterDescriptor(KubernetesClusterClientFactory.java:39)
    at org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:63)
    at org.apache.flink.client.cli.CliFrontend.runApplication(CliFrontend.java:213)
    at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1057)
    at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
    at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
    at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
</code></pre>
<p>After reading</p>
<ul>
<li><a href="https://www.mail-archive.com/search?l=dev@spark.apache.org&amp;q=subject:%22Should+we+add+built+in+support+for+bouncy+castle+EC+w%5C%2FKube%22&amp;o=newest&amp;f=1" rel="nofollow noreferrer">https://www.mail-archive.com/search?l=dev@spark.apache.org&amp;q=subject:&quot;Should+we+add+built+in+support+for+bouncy+castle+EC+w\%2FKube&quot;&amp;o=newest&amp;f=1</a></li>
<li><a href="https://github.com/de-jcup/ekube/issues/63#issuecomment-753508790" rel="nofollow noreferrer">https://github.com/de-jcup/ekube/issues/63#issuecomment-753508790</a></li>
</ul>
<p>I added</p>
<pre><code>&lt;dependency&gt;
    &lt;groupId&gt;org.bouncycastle&lt;/groupId&gt;
    &lt;artifactId&gt;bcpkix-jdk15on&lt;/artifactId&gt;
    &lt;version&gt;1.69&lt;/version&gt;
&lt;/dependency&gt;
&lt;dependency&gt;
    &lt;groupId&gt;org.bouncycastle&lt;/groupId&gt;
    &lt;artifactId&gt;bcprov-jdk15on&lt;/artifactId&gt;
    &lt;version&gt;1.69&lt;/version&gt;
&lt;/dependency&gt;
&lt;dependency&gt;
    &lt;groupId&gt;org.bouncycastle&lt;/groupId&gt;
    &lt;artifactId&gt;bcprov-ext-jdk15on&lt;/artifactId&gt;
    &lt;version&gt;1.69&lt;/version&gt;
&lt;/dependency&gt;
</code></pre>
<p>to my <strong>pom.xml</strong> file. I built and pushed to Docker Hub again.</p>
<p>When I ran the Flink command above, I still got same error. Any idea? Thanks!</p>
<hr />
<p><strong>UPDATE 1:</strong></p>
<p>Besides above <strong>pom.xml</strong> change, I manually downloaded those 3 jars and changed my Dockerfile to</p>
<pre><code>FROM flink
COPY lib/* $FLINK_HOME/lib
RUN mkdir -p $FLINK_HOME/usrlib
COPY target/streaming-0.1.jar $FLINK_HOME/usrlib/streaming-0.1.jar
</code></pre>
<p>and tried again, but still same error.</p>
<p>I can confirm the 3 jar files <code>bcpkix-jdk15on-1.69.jar</code>, <code>bcprov-ext-jdk15on-1.69.jar</code>, <code>bcprov-jdk15on-1.69.jar</code> are in the docker image:</p>
<pre><code>➜ docker run -it 6c48af48db55c334003a307d1ef7a5fc5181f389613284b66b5cb97588b9708d sh

$ cd lib &amp;&amp; ls
bcpkix-jdk15on-1.69.jar      flink-dist_2.12-1.13.2.jar     flink-table_2.12-1.13.2.jar  log4j-slf4j-impl-2.12.1.jar
bcprov-ext-jdk15on-1.69.jar  flink-json-1.13.2.jar      log4j-1.2-api-2.12.1.jar
bcprov-jdk15on-1.69.jar      flink-shaded-zookeeper-3.4.14.jar  log4j-api-2.12.1.jar
flink-csv-1.13.2.jar         flink-table-blink_2.12-1.13.2.jar  log4j-core-2.12.1.jar
$ cd ../usrlib &amp;&amp; ls
streaming-0.1.jar
</code></pre>
<hr />
<p><strong>UPDATE 2:</strong></p>
<p>I tried to start the session mode by</p>
<pre><code>/usr/local/Cellar/apache-flink/1.13.1/libexec/bin/kubernetes-session.sh
</code></pre>
<p>but still got same error. So now I can confirm when I was using application mode before, the issue is not related my Docker image.</p>
<p>I have those jars are located at <code>~/.m2</code> on my machine:</p>
<p><a href="https://i.stack.imgur.com/hQNf8.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/hQNf8.png" alt="enter image description here" /></a></p>
<p>Did I miss any other jars?</p>
<p>Also, I found the error only happens to the cluster created by k3d/k3s, but not minikube.</p>

<h1>TL;DR</h1>
<p>Download latest version of <code>bcprov-jdk15on</code> and <code>bcpkix-jdk15on</code> jar files from</p>
<ul>
<li><a href="https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on" rel="nofollow noreferrer">https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on</a></li>
<li><a href="https://mvnrepository.com/artifact/org.bouncycastle/bcpkix-jdk15on" rel="nofollow noreferrer">https://mvnrepository.com/artifact/org.bouncycastle/bcpkix-jdk15on</a></li>
</ul>
<p>Then move to your apache-flink folder. In my case, it is</p>
<pre><code>/usr/local/Cellar/apache-flink/{version}/libexec/lib
</code></pre>
<p>Then you are good to go!</p>
<hr />
<h1>How I Found Out</h1>
<p>After checking the code of</p>
<ul>
<li>/usr/local/Cellar/apache-flink/1.13.1/libexec/bin/kubernetes-session.sh</li>
<li>/usr/local/Cellar/apache-flink/1.13.1/libexec/libexec/kubernetes-session.sh</li>
</ul>
<p>The first script is pointing to the second script, and the second script has</p>
<pre><code># ...
CC_CLASSPATH=`manglePathList $(constructFlinkClassPath):$INTERNAL_HADOOP_CLASSPATHS`

# ...
&quot;$JAVA_RUN&quot; $JVM_ARGS -classpath &quot;$CC_CLASSPATH&quot; $log_setting org.apache.flink.kubernetes.cli.KubernetesSessionCli &quot;$@&quot;
</code></pre>
<p>I added <code>echo $CC_CLASSPATH</code>, and printed out the classpath.</p>
<p>In my case, it is at <code>/usr/local/Cellar/apache-flink/1.13.1/libexec/lib</code>.</p>
<p>After I put <code>bcprov-jdk15on-1.69.jar</code> and <code>bcpkix-jdk15on-1.69.jar</code> in the folder above, Flink can be deployed to k3s (k3d) now in both session and application modes.</p>
