title: How to use local Docker images when submitting Spark jobs (2.3) natively to Kubernetes?
body: <p>I am trying to submit a Spark Job on Kubernetes natively using Apache Spark 2.3.
  When I use a Docker image on Docker Hub (for Spark 2.2), it works:</p>
  
  <pre><code>bin/spark-submit \
      --master k8s://http://localhost:8080 \
      --deploy-mode cluster \
      --name spark-pi \
      --class org.apache.spark.examples.SparkPi \
      --conf spark.executor.instances=5 \
      --conf spark.kubernetes.container.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 \
      local:///home/fedora/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar 
  </code></pre>
  
  <p>However, when I try to build a local Docker image,</p>
  
  <pre><code>sudo docker build -t spark:2.3 -f kubernetes/dockerfiles/spark/Dockerfile .
  </code></pre>
  
  <p>and submit the job as:</p>
  
  <pre><code>bin/spark-submit \
  --master k8s://http://localhost:8080 \
  --deploy-mode cluster \
  --name spark-pi \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.executor.instances=5 \
  --conf spark.kubernetes.container.image=spark:2.3 \
  local:///home/fedora/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar
  </code></pre>
  
  <p>I get the following error; <strong>that is "repository docker.io/spark not found: does not exist or no pull access, reason=ErrImagePull, additionalProperties={})"</strong></p>
  
  <pre><code>status: [ContainerStatus(containerID=null, image=spark:2.3, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=rpc error: code = 2 desc = repository docker.io/spark not found: does not exist or no pull access, reason=ErrImagePull, additionalProperties={}), additionalProperties={}), additionalProperties={})]
  2018-03-15 11:09:54 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
       pod name: spark-pi-3a1a6e8ce615395fa7df81eac06d58ed-driver
       namespace: default
       labels: spark-app-selector -&gt; spark-8d9fdaba274a4eb69e28e2a242fe86ca, spark-role -&gt; driver
       pod uid: 5271602b-2841-11e8-a78e-fa163ed09d5f
       creation time: 2018-03-15T11:09:25Z
       service account name: default
       volumes: default-token-v4vhk
       node name: mlaas-p4k3djw4nsca-minion-1
       start time: 2018-03-15T11:09:25Z
       container images: spark:2.3
       phase: Pending
       status: [ContainerStatus(containerID=null, image=spark:2.3, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=Back-off pulling image "spark:2.3", reason=ImagePullBackOff, additionalProperties={}), additionalProperties={}), additionalProperties={})]
  </code></pre>
  
  <p>Also, I tried to run a local Docker registry as described in:
  <a href="https://docs.docker.com/registry/deploying/#run-a-local-registry" rel="nofollow noreferrer">https://docs.docker.com/registry/deploying/#run-a-local-registry</a></p>
  
  <pre><code>docker run -d -p 5000:5000 --restart=always --name registry registry:2
  
  sudo docker tag spark:2.3 localhost:5000/spark:2.3
  
  sudo docker push localhost:5000/spark:2.3
  </code></pre>
  
  <p>I can do this successfully:
  docker pull localhost:5000/spark:2.3</p>
  
  <p>However, when I submit the Spark job:</p>
  
  <pre><code>bin/spark-submit \
      --master k8s://http://localhost:8080 \
      --deploy-mode cluster \
      --name spark-pi \
      --class org.apache.spark.examples.SparkPi \
      --conf spark.executor.instances=5 \
      --conf spark.kubernetes.container.image=localhost:5000/spark:2.3 \
      local:///home/fedora/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar
  </code></pre>
  
  <p>I again got ErrImagePull:</p>
  
  <pre><code>status: [ContainerStatus(containerID=null, image=localhost:5000/spark:2.3, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=rpc error: code = 2 desc = Error while pulling image: Get http://localhost:5000/v1/repositories/spark/images: dial tcp [::1]:5000: getsockopt: connection refused, reason=ErrImagePull, additionalProperties={}), additionalProperties={}), additionalProperties={})]
  </code></pre>
  
  <p>Is there a way in Spark 2.3 to use local Docker images when submitting jobs natively to Kubernetes?</p>
  
  <p>Thank you in advance.</p>