<p>I am trying to follow <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#cluster-mode" rel="nofollow noreferrer">this Spark document</a> to use the cluster mode.</p>
<p>I deployed Spark in a local Kubernetes at namespace <code>hm-spark</code> by</p>
<pre class="lang-bash prettyprint-override"><code>helm upgrade \
  spark \
  spark \
  --install \
  --repo=https://charts.bitnami.com/bitnami \
  --namespace=hm-spark \
  --create-namespace \
  --values=my-values.yaml
</code></pre>
<p><strong>my-values.yaml</strong></p>
<pre class="lang-yaml prettyprint-override"><code>image:
  registry: docker.io
  repository: bitnami/spark
  tag: 3.4.0-debian-11-r1
</code></pre>
<p>I got Kubernetes IP <code>https://127.0.0.1:6443</code> by</p>
<pre><code>➜ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
</code></pre>
<p>Then when I submit the Spark application on my macOS by:</p>
<pre><code>spark-submit \
    --master=k8s://https://127.0.0.1:6443 \
    --deploy-mode=cluster \
    --name=spark-pi \
    --class=org.apache.spark.examples.SparkPi \
    --conf=spark.kubernetes.namespace=hm-spark \
    --conf=spark.kubernetes.container.image=docker.io/bitnami/spark:3.4.0-debian-11-r1 \
    local:///opt/bitnami/spark/examples/jars/spark-examples_2.12-3.4.0.jar
</code></pre>
<p>The <code>spark-pi</code> pod got created</p>
<pre><code>➜ kubectl get pods --namespace hm-spark
NAME                                                READY   STATUS      RESTARTS        AGE
spark-worker-0                                      1/1     Running     0               82m
spark-worker-1                                      1/1     Running     0               82m
spark-master-0                                      1/1     Running     0               82m
spark-pi-ec6d2e886f483472-driver                    0/1     Error       0               9m32s
</code></pre>
<p>However, it failed with error:</p>
<pre class="lang-bash prettyprint-override"><code>spark 00:50:00.00 
spark 00:50:00.01 Welcome to the Bitnami spark container
spark 00:50:00.01 Subscribe to project updates by watching https://github.com/bitnami/containers
spark 00:50:00.01 Submit issues and feature requests at https://github.com/bitnami/containers/issues
spark 00:50:00.01 

23/05/31 00:50:02 INFO SparkContext: Running Spark version 3.4.0
23/05/31 00:50:02 INFO ResourceUtils: ==============================================================
23/05/31 00:50:02 INFO ResourceUtils: No custom resources configured for spark.driver.
23/05/31 00:50:02 INFO ResourceUtils: ==============================================================
23/05/31 00:50:02 INFO SparkContext: Submitted application: Spark Pi
23/05/31 00:50:02 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(memory -&gt; name: memory, amount: 1024, script: , vendor: , offHeap -&gt; name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -&gt; name: cpus, amount: 1.0)
23/05/31 00:50:02 INFO ResourceProfile: Limiting resource is cpu
23/05/31 00:50:02 INFO ResourceProfileManager: Added ResourceProfile id: 0
23/05/31 00:50:02 INFO SecurityManager: Changing view acls to: spark
23/05/31 00:50:02 INFO SecurityManager: Changing modify acls to: spark
23/05/31 00:50:02 INFO SecurityManager: Changing view acls groups to: 
23/05/31 00:50:02 INFO SecurityManager: Changing modify acls groups to: 
23/05/31 00:50:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: spark; groups with view permissions: EMPTY; users with modify permissions: spark; groups with modify permissions: EMPTY
23/05/31 00:50:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/05/31 00:50:02 INFO Utils: Successfully started service 'sparkDriver' on port 7078.
23/05/31 00:50:02 INFO SparkEnv: Registering MapOutputTracker
23/05/31 00:50:02 INFO SparkEnv: Registering BlockManagerMaster
23/05/31 00:50:02 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
23/05/31 00:50:02 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
23/05/31 00:50:02 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
23/05/31 00:50:02 INFO DiskBlockManager: Created local directory at /var/data/spark-a1bd571d-599f-48d6-b7a9-06d35fb82cdb/blockmgr-5ee7264e-b52a-48bc-a1bf-f8f3f6a514aa
23/05/31 00:50:02 INFO MemoryStore: MemoryStore started with capacity 413.9 MiB
23/05/31 00:50:02 INFO SparkEnv: Registering OutputCommitCoordinator
23/05/31 00:50:02 INFO JettyUtils: Start Jetty 0.0.0.0:4040 for SparkUI
23/05/31 00:50:02 INFO Utils: Successfully started service 'SparkUI' on port 4040.
23/05/31 00:50:02 INFO SparkContext: Added JAR local:///opt/bitnami/spark/examples/jars/spark-examples_2.12-3.4.0.jar at file:/opt/bitnami/spark/examples/jars/spark-examples_2.12-3.4.0.jar with timestamp 1685494202182
23/05/31 00:50:02 WARN SparkContext: The JAR local:///opt/bitnami/spark/examples/jars/spark-examples_2.12-3.4.0.jar at file:/opt/bitnami/spark/examples/jars/spark-examples_2.12-3.4.0.jar has been added already. Overwriting of added jar is not supported in the current version.
23/05/31 00:50:02 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://spark-master:7077...
23/05/31 00:50:03 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master spark-master:7077
org.apache.spark.SparkException: Exception thrown in awaitResult: 
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:322)
    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
    at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:102)
    at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:110)
    at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anon$1.run(StandaloneAppClient.scala:108)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.IOException: Failed to connect to spark-master/&lt;unresolved&gt;:7077
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:284)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:214)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:226)
    at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
    ... 4 more
Caused by: java.net.UnknownHostException: spark-master
    at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
    at java.base/java.net.InetAddress.getByName(InetAddress.java:1256)
    at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156)
    at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
    at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153)
    at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41)
    at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61)
    at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53)
    at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55)
    at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31)
    at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106)
    at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:206)
    at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:46)
    at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:180)
    at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:166)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
    at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
    at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
    at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
    at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:990)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:516)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
    at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486)
    at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    ... 1 more
# ...
23/05/31 00:51:02 WARN StandaloneSchedulerBackend: Application ID is not initialized yet.
23/05/31 00:51:02 ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
23/05/31 00:51:02 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 7079.
23/05/31 00:51:02 INFO NettyBlockTransferService: Server created on spark-pi-ec6d2e886f483472-driver-svc.hm-spark.svc:7079
23/05/31 00:51:02 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
23/05/31 00:51:02 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, spark-pi-ec6d2e886f483472-driver-svc.hm-spark.svc, 7079, None)
23/05/31 00:51:02 INFO BlockManagerMasterEndpoint: Registering block manager spark-pi-ec6d2e886f483472-driver-svc.hm-spark.svc:7079 with 413.9 MiB RAM, BlockManagerId(driver, spark-pi-ec6d2e886f483472-driver-svc.hm-spark.svc, 7079, None)
23/05/31 00:51:02 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, spark-pi-ec6d2e886f483472-driver-svc.hm-spark.svc, 7079, None)
23/05/31 00:51:02 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, spark-pi-ec6d2e886f483472-driver-svc.hm-spark.svc, 7079, None)
23/05/31 00:51:03 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
23/05/31 00:51:03 INFO SparkContext: Starting job: reduce at SparkPi.scala:38
23/05/31 00:51:03 INFO SparkContext: SparkContext is stopping with exitCode 0.
23/05/31 00:51:03 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
23/05/31 00:51:03 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
23/05/31 00:51:03 INFO DAGScheduler: Parents of final stage: List()
23/05/31 00:51:03 INFO DAGScheduler: Missing parents: List()
23/05/31 00:51:03 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
23/05/31 00:51:03 INFO SparkUI: Stopped Spark web UI at http://spark-pi-ec6d2e886f483472-driver-svc.hm-spark.svc:4040
23/05/31 00:51:03 INFO TaskSchedulerImpl: Cancelling stage 0
23/05/31 00:51:03 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage cancelled
23/05/31 00:51:03 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) failed in 0.028 s due to Job aborted due to stage failure: Task serialization failed: java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
This stopped SparkContext was created at:

org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1020)
org.apache.spark.examples.SparkPi$.main(SparkPi.scala:30)
org.apache.spark.examples.SparkPi.main(SparkPi.scala)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.base/java.lang.reflect.Method.invoke(Method.java:568)
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
</code></pre>
<p>And here is my Kubernetes services:</p>
<pre class="lang-bash prettyprint-override"><code>➜ kubectl get services --namespace hm-spark
NAME                                                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
spark-headless                                                 ClusterIP   None            &lt;none&gt;        &lt;none&gt;                       82m
spark-master-svc                                               ClusterIP   10.43.164.158   &lt;none&gt;        7077/TCP,80/TCP              82m
spark-pi-ec6d2e886f483472-driver-svc                           ClusterIP   None            &lt;none&gt;        7078/TCP,7079/TCP,4040/TCP   10m
</code></pre>
<p>Any idea? Thanks!</p>

Hongbo Miao
<p>I have no specific experience with these Bitnami helm charts, but it seems to me like your application is both trying to use a:</p>
<ul>
<li><a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#cluster-mode" rel="nofollow noreferrer">Kubernetes cluster manager</a> (judging from the URL in your <code>--master</code> config, starting with <code>k8s://</code>)</li>
<li>and a <a href="https://spark.apache.org/docs/latest/spark-standalone.html" rel="nofollow noreferrer">Standalone cluster manager</a> (judging from the error messages, port 7077 and the <code>spark://</code> part of the master URL)</li>
</ul>
<p>That seems like a bit of a mix up: you should choose between one of both. After having a look at some docs around those Bitnami helm charts, I found <a href="https://github.com/bitnami/charts/tree/main/bitnami/spark#submit-an-application" rel="nofollow noreferrer">this example</a>:</p>
<pre class="lang-bash prettyprint-override"><code>$ ./bin/spark-submit \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.kubernetes.container.image=bitnami/spark:3 \
    --master k8s://https://k8s-apiserver-host:k8s-apiserver-port \
    --conf spark.kubernetes.driverEnv.SPARK_MASTER_URL=spark://spark-master-svc:spark-master-port \
    --deploy-mode cluster \
    ./examples/jars/spark-examples_2.12-3.2.0.jar 1000
</code></pre>
<p>Again, I'm not entirely familiar with these helm charts but it seems like you might be missing a critical configuration bit concerning which master will finally be used, namely:</p>
<pre><code>--conf spark.kubernetes.driverEnv.SPARK_MASTER_URL=spark://spark-master-svc:spark-master-port
</code></pre>
