<p>I am trying to build spark image to use with Kubernetes (minikube locally) (following <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html" rel="nofollow noreferrer">https://spark.apache.org/docs/latest/running-on-kubernetes.html</a>, <a href="https://itnext.io/running-spark-job-on-kubernetes-minikube-958cadaddd55" rel="nofollow noreferrer">https://itnext.io/running-spark-job-on-kubernetes-minikube-958cadaddd55</a> and <a href="https://learn.microsoft.com/en-us/azure/aks/spark-job" rel="nofollow noreferrer">https://learn.microsoft.com/en-us/azure/aks/spark-job</a>).
I want to build it for local docker repo and I am using following command <code>./bin/docker-image-tool.sh -m -t v2 build</code> to build it for minikube. This build succeeds, but after that I can't find the image with <code>docker image ps</code>. The spark-submit fails with cryptic <code>Error: Could not find or load main class org.apache.spark.launcher.Main</code>, which doesn't tell anything, but it seems reasonable to be linked with image missing in the docker registry.</p>
<p>All <em>build</em> steps seem to succeed with no issue, so I am bit lost at what went wrong.</p>
<p>Build output:</p>
<pre><code>Sending build context to Docker daemon  1.193GB
Step 1/15 : FROM openjdk:8-jdk-slim
 ---&gt; 6b2212023a6e
Step 2/15 : ARG spark_jars=jars
 ---&gt; Using cache
 ---&gt; 2407ea1f6efa
Step 3/15 : ARG img_path=kubernetes/dockerfiles
 ---&gt; Using cache
 ---&gt; db7404f15dfd
Step 4/15 : ARG k8s_tests=kubernetes/tests
 ---&gt; Using cache
 ---&gt; 8dc19efc9230
Step 5/15 : RUN set -ex &amp;&amp;     apt-get update &amp;&amp;     ln -s /lib /lib64 &amp;&amp;     apt install -y bash tini libc6 libpam-modules libnss3 &amp;&amp;     mkdir -p /opt/spark &amp;&amp;     mkdir -p /opt/spark/work-dir &amp;&amp;     touch /opt/spark/RELEASE &amp;&amp;     rm /bin/sh &amp;&amp;     ln -sv /bin/bash /bin/sh &amp;&amp;     echo &quot;auth required pam_wheel.so use_uid&quot; &gt;&gt; /etc/pam.d/su &amp;&amp;     chgrp root /etc/passwd &amp;&amp; chmod ug+rw /etc/passwd &amp;&amp;     rm -rf /var/cache/apt/*
 ---&gt; Using cache
 ---&gt; 5884173cd551
Step 6/15 : COPY ${spark_jars} /opt/spark/jars
 ---&gt; Using cache
 ---&gt; ff058e0c80e7
Step 7/15 : COPY bin /opt/spark/bin
 ---&gt; Using cache
 ---&gt; d158732d730c
Step 8/15 : COPY sbin /opt/spark/sbin
 ---&gt; Using cache
 ---&gt; 896cd481cdbc
Step 9/15 : COPY ${img_path}/spark/entrypoint.sh /opt/
 ---&gt; Using cache
 ---&gt; 1d108a1e1df4
Step 10/15 : COPY examples /opt/spark/examples
 ---&gt; Using cache
 ---&gt; 3a7680f04b03
Step 11/15 : COPY ${k8s_tests} /opt/spark/tests
 ---&gt; Using cache
 ---&gt; 2cd76b303d3e
Step 12/15 : COPY data /opt/spark/data
 ---&gt; Using cache
 ---&gt; 8221e20aac95
Step 13/15 : ENV SPARK_HOME /opt/spark
 ---&gt; Using cache
 ---&gt; ecb9bee32447
Step 14/15 : WORKDIR /opt/spark/work-dir
 ---&gt; Using cache
 ---&gt; 6bfafec8d0bb
Step 15/15 : ENTRYPOINT [ &quot;/opt/entrypoint.sh&quot; ]
 ---&gt; Using cache
 ---&gt; 110ccecb98db
Successfully built 110ccecb98db
Successfully tagged spark:v2
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.
Sending build context to Docker daemon  1.193GB
Step 1/9 : ARG base_img
Step 2/9 : FROM $base_img
 ---&gt; 110ccecb98db
Step 3/9 : WORKDIR /
 ---&gt; Using cache
 ---&gt; 7d82c43a71b8
Step 4/9 : RUN mkdir ${SPARK_HOME}/python
 ---&gt; Using cache
 ---&gt; a06db167de65
Step 5/9 : RUN apt install -y python python-pip &amp;&amp;     apt install -y python3 python3-pip &amp;&amp;     rm -r /usr/lib/python*/ensurepip &amp;&amp;     pip install --upgrade pip setuptools &amp;&amp;     rm -r /root/.cache &amp;&amp; rm -rf /var/cache/apt/*
 ---&gt; Using cache
 ---&gt; 2789da56886a
Step 6/9 : COPY python/lib ${SPARK_HOME}/python/lib
 ---&gt; Using cache
 ---&gt; 17ad5567580c
Step 7/9 : ENV PYTHONPATH ${SPARK_HOME}/python/lib/pyspark.zip:${SPARK_HOME}/python/lib/py4j-*.zip
 ---&gt; Using cache
 ---&gt; 01f0ee91d6a6
Step 8/9 : WORKDIR /opt/spark/work-dir
 ---&gt; Using cache
 ---&gt; 95b4885abc7e
Step 9/9 : ENTRYPOINT [ &quot;/opt/entrypoint.sh&quot; ]
 ---&gt; Using cache
 ---&gt; 2bb9c23681e8
Successfully built 2bb9c23681e8
Successfully tagged spark-py:v2
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.
Sending build context to Docker daemon  1.193GB
Step 1/9 : ARG base_img
Step 2/9 : FROM $base_img
 ---&gt; 110ccecb98db
Step 3/9 : WORKDIR /
 ---&gt; Using cache
 ---&gt; 7d82c43a71b8
Step 4/9 : RUN mkdir ${SPARK_HOME}/R
 ---&gt; Using cache
 ---&gt; c49d10da5c74
Step 5/9 : RUN apt install -y r-base r-base-dev &amp;&amp; rm -rf /var/cache/apt/*
 ---&gt; Using cache
 ---&gt; d693ef1ab12c
Step 6/9 : COPY R ${SPARK_HOME}/R
 ---&gt; Using cache
 ---&gt; 50ca4b703501
Step 7/9 : ENV R_HOME /usr/lib/R
 ---&gt; Using cache
 ---&gt; cba4fbc51e7c
Step 8/9 : WORKDIR /opt/spark/work-dir
 ---&gt; Using cache
 ---&gt; 449edacaae16
Step 9/9 : ENTRYPOINT [ &quot;/opt/entrypoint.sh&quot; ]
 ---&gt; Using cache
 ---&gt; a84c327a2b9d
Successfully built a84c327a2b9d
Successfully tagged spark-r:v2
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.
</code></pre>

<p>I have missed these as well. But this should be OK</p>
<p>You see the lines</p>
<pre><code>Successfully tagged spark:v2
Successfully tagged spark-py:v2
Successfully tagged spark-r:v2
</code></pre>
<p>so your build has succeeded, and the Docker images for JVM-, Python- and R-based jobs to be used for Spark jobs on Kubernetes were created.</p>
<p>You will <em><strong>not</strong></em> see these images when you list the images <em><strong>on the host</strong></em> (the system <em>minikube</em> runs on):</p>
<pre><code>$ docker image ls | egrep &quot;spark-k8s-2.4.5|minikube&quot;
gcr.io/k8s-minikube/kicbase                                          v0.0.18                                         a776c544501a        6 weeks ago         1.08GB
</code></pre>
<p>You should see these images when you list the images <em><strong>in minikube</strong></em>:</p>
<pre><code>$ minikube ssh
Last login: Thu Apr  8 15:55:10 2021 from 192.168.49.1
docker@minikube:~$ docker image ls
REPOSITORY                                TAG               IMAGE ID       CREATED         SIZE
spark-r                                   spark-k8s-2.4.5   21836f5f1e0c   23 hours ago    1.1GB
spark-py                                  spark-k8s-2.4.5   1a042cd867f7   23 hours ago    1.05GB
spark                                     spark-k8s-2.4.5   07a9a8459c4c   23 hours ago    552MB
openjdk                                   8-jdk-slim        76fad6e75a0f   9 days ago      288MB
k8s.gcr.io/kube-proxy                     v1.20.2           43154ddb57a8   2 months ago    118MB
k8s.gcr.io/kube-apiserver                 v1.20.2           a8c2fdb8bf76   2 months ago    122MB
k8s.gcr.io/kube-controller-manager        v1.20.2           a27166429d98   2 months ago    116MB
k8s.gcr.io/kube-scheduler                 v1.20.2           ed2c44fbdd78   2 months ago    46.4MB
kubernetesui/dashboard                    v2.1.0            9a07b5b4bfac   3 months ago    226MB
gcr.io/k8s-minikube/storage-provisioner   v4                85069258b98a   4 months ago    29.7MB
k8s.gcr.io/etcd                           3.4.13-0          0369cf4303ff   7 months ago    253MB
k8s.gcr.io/coredns                        1.7.0             bfe3a36ebd25   9 months ago    45.2MB
kubernetesui/metrics-scraper              v1.0.4            86262685d9ab   12 months ago   36.9MB
k8s.gcr.io/pause                          3.2               80d28bedfe5d   13 months ago   683kB
k8s.gcr.io/echoserver                     1.4               a90209bb39e3   4 years ago     140MB
</code></pre>
<p>This is the effect of using</p>
<pre><code>docker-image-tool.sh -m
</code></pre>
<p><em>-m</em> instructs <em>docker-image-tool.sh</em> to &quot;use minikube's Docker daemon&quot;.</p>
<p>Likewise <em>docker ps</em> on the host will look like this: It will show <em>minikube</em></p>
<pre><code>$ docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS                  PORTS                                                                                                                                  NAMES
3cb659bf50d7        gcr.io/k8s-minikube/kicbase:v0.0.18   &quot;/usr/local/bin/entr…&quot;   8 days ago          Up 56 minutes           127.0.0.1:32782-&gt;22/tcp, 127.0.0.1:32781-&gt;2376/tcp, 127.0.0.1:32780-&gt;5000/tcp, 127.0.0.1:32779-&gt;8443/tcp, 127.0.0.1:32778-&gt;32443/tcp   minikube
</code></pre>
<p>O the other hand you should see various Kubernetes services in <em>minikube</em> like this:</p>
<pre><code>$ minikube ssh
Last login: Fri Apr  9 06:59:08 2021 from 192.168.49.1
docker@minikube:~$ docker ps | head -n 5
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS     NAMES
43b8fa28c0f0   9a07b5b4bfac           &quot;/dashboard --insecu…&quot;   56 minutes ago   Up 56 minutes             k8s_kubernetes-dashboard_kubernetes-dashboard-968bcb79-hnqcg_kubernetes-dashboard_90252423-eb5d-4e53-800a-3aa40cbcd4b6_9
3cc6d85f88ce   85069258b98a           &quot;/storage-provisioner&quot;   56 minutes ago   Up 56 minutes             k8s_storage-provisioner_storage-provisioner_kube-system_ca8a31c4-1ca2-4284-b9d4-4a1068d46876_15
cf48ca7b8e2e   bfe3a36ebd25           &quot;/coredns -conf /etc…&quot;   57 minutes ago   Up 57 minutes             k8s_coredns_coredns-74ff55c5b-xtkgf_kube-system_d24f114f-d3f9-4a45-bd2a-fa954a42686e_7
0ce17fa229df   a90209bb39e3           &quot;nginx -g 'daemon of…&quot;   57 minutes ago   Up 57 minutes             k8s_echoserver_hello-minikube-6ddfcc9757-vwkh8_default_08c3f2ee-c0ed-4778-9aa9-5d65890cb18c_7
</code></pre>
<p>As for the error message I don't think that it's related to the &quot;missing&quot; image.
It could be anything, like invalid class name, the class is not the jar, dependency problems (for example in the jar file containing the job), mixed Spark versions etc.</p>
