<p>I have 3 node[host a,host b, host c] kubernetes cluster(version 1.12.2). I am trying run spark-pi example jar as mentioned in <a href="https://kubernetes.io/blog/2018/03/apache-spark-23-with-native-kubernetes/" rel="nofollow noreferrer">kubernetes document</a>.</p>

<p>Host a is my kubernetes Master. >> kubectl get nodees list all the three nodes.</p>

<p>I have built the spark docker image using whats provided in spark 2.3.0 binary folder.</p>

<pre><code>&gt;&gt; sudo ./bin/docker-image-tool.sh -r docker.io/spark/spark -t spark230 build
</code></pre>

<p>I got the message the image got built successfully.</p>

<pre><code>&gt;&gt; docker images ls

REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
docker.io/spark/spark                spark230            6a2b645d7efe        About an hour ago   346 MB
docker.io/weaveworks/weave-npc       2.5.0               d499500e93d3        7 days ago          49.5 MB
docker.io/weaveworks/weave-kube      2.5.0               a5103f96993a        7 days ago          148 MB
docker.io/openjdk                    8-alpine            97bc1352afde        2 weeks ago         103 MB
k8s.gcr.io/kube-proxy                v1.12.2             15e9da1ca195        2 weeks ago         96.5 MB
k8s.gcr.io/kube-apiserver            v1.12.2             51a9c329b7c5        2 weeks ago         194 MB
k8s.gcr.io/kube-controller-manager   v1.12.2             15548c720a70        2 weeks ago         164 MB
k8s.gcr.io/kube-scheduler            v1.12.2             d6d57c76136c        2 weeks ago         58.3 MB
k8s.gcr.io/etcd                      3.2.24              3cab8e1b9802        7 weeks ago         220 MB
k8s.gcr.io/coredns                   1.2.2               367cdc8433a4        2 months ago        39.2 MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        10 months ago       742 kB
</code></pre>



<pre><code>&gt; ./bin/spark-submit     
 --master k8s://https://&lt;api-server&gt;:&lt;api
&gt; server port&gt;      --deploy-mode cluster      --name spark-pi     
&gt; --class org.apache.spark.examples.SparkPi       --conf spark.executor.instances=5     --conf
&gt; spark.kubernetes.container.image=spark/spark:spark230    --conf
&gt; spark.kubernetes.authenticate.driver.serviceAccountName=spark   
&gt; local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar
</code></pre>

<p>When I submit above command , it gives proper output sometimes. Other times it throws below error. </p>

<pre><code>&gt;  code = Unknown desc = repository docker.io/spark/spark not found:
 does not exist or no pull access, reason=ErrImagePull
</code></pre>

<p>When I debugged it further, it looks like, whenever  node name: host b or host c its throwing above error message.</p>

<p>When node name : host a , then it runs fine. Looks like other nodes are unable to locate the image. </p>

<p>Questions: </p>

<ol>
<li>Should I install spark on all nodes &amp; build docker image on all nodes? </li>
<li>Is it possible to pass the image reference from single node [host a] to other nodes? i.e how to make other nodes refer the same image from host a.</li>
</ol>

<p>Yes, you need to build Spark image in all the nodes. You can write the wrapper script to invoke the 'rebuild-image.sh' on all the nodes as below:</p>

<pre><code>for h in hostnames; do
    rsync -av /opt/spark ${h}:/opt
    ssh ${h} /opt/spark/rebuild-image.sh
</code></pre>
