<p>I am trying to use <code>spark-submit</code> with <code>client</code> mode in the kubernetes pod to submit jobs to EMR (Due to some other infra issues, we don't allow <code>cluster</code> mode). 
By default, <code>spark-submit</code> uses the <code>hostname</code> of the pod as the <code>spark.driver.host</code> and the <code>hostname</code> is the pod's hostname so <code>spark executor</code> could not resolve it. And the <code>spark.driver.port</code> is also locally to the pod (container). </p>

<p>I know a way to pass some confs to <code>spark-submit</code> so that the <code>spark executor</code> can talk to the <code>driver</code>, those configs are: </p>

<p><code>--conf spark.driver.bindAddress=0.0.0.0 --conf spark.driver.host=$HOST_IP_OF_K8S_WORKER --conf spark.driver.port=32000 --conf spark.driver.blockManager.port=32001</code></p>

<p>and create a service to in the kubernetes so that <code>spark executor</code> can talk to the <code>driver</code>:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: spark-block-manager
  namespace: my-app
spec:
  selector:
    app: my-app
  type: NodePort
  ports:
    - name: port-0
      nodePort: 32000
      port: 32000
      protocol: TCP
      targetPort: 32000
    - name: port-1
      nodePort: 32001
      port: 32001
      protocol: TCP
      targetPort: 32001
    - name: port-2
      nodePort: 32002
      port: 32002
      protocol: TCP
      targetPort: 32002
</code></pre>

<hr>

<p>But the issue is there are can be more than 1 pods running on one k8s worker and even more than 1 <code>spark-submit</code> jobs in one pod. So before launching a pod, we need to dynamically select few available ports in the k8s node and create a service to do the port mapping and then during launching the pod, pass those ports into the pod to tell <code>spark-submit</code> to use them. I feel like this is a little bit complex.</p>

<p>Using <code>hostNetwork: true</code> could potentially solve this issue but it introduces lots of other issues in our infra so this is not an option.</p>

<p>If <code>spark-submit</code> can support <code>bindPort</code> concept just like <code>driver.bindAddress</code> and <code>driver.host</code> or support <code>proxy</code>, it will be cleaner to solve the issue.</p>

<p>Does someone have similar situation? Please share some insights.</p>

<p>Thanks.</p>

<p>Additional context:
<code>spark version</code>: <code>2.4</code></p>

<p>Spark submit can take additional args like, <code>--conf spark.driver.bindAddress, --conf spark.driver.host, --conf spark.driver.port, --conf spark.driver.blockManager.port, --conf spark.port.maxRetries</code>. The <code>spark.driver.host</code> and <code>driver.port</code> is used to tell Spark Executor to use this host and port to connect back to the Spark submit.</p>

<p>We use <code>hostPort</code> and <code>containerPort</code> to expose the ports inside the container, inject the port range and <code>hostIP</code> as the environment variables to the Pod so that spark-submit knows what to use. So those additional args are:</p>

<pre><code>--conf spark.driver.bindAddress=0.0.0.0` # has to be `0.0.0.0` so that it is accessible outside pod
--conf spark.driver.host=$HOST_IP # k8s worker ip, can be easily injected to the pod
--conf spark.driver.port=$SPARK_DRIVER_PORT # defined as environment variable
--conf spark.driver.blockManager.port=$SPARK_DRIVER_PORT # defined as environment variable
--conf spark.port.maxRetries=$SPARK_PORT_MAX_RETRIES # defined as environment variable
</code></pre>

<p>The <code>hostPort</code> is local to the Kubernetes worker, which means we don’t need to worry about the run out of ports. The k8s scheduler can find a host to run the pod.</p>

<p>We can reserve the ports from 40000 to 49000 on each host, and open 8 ports for each pod (as each spark-submit requires 2 open ports). The ports are chosen based on the pod_id. Since Kubernetes recommends running less than 100 pods per node, the ports collision will be very rare.</p>
