<p>We are running Spark 3.1.1 on Kubernetes in client mode.</p>
<p>We are a simple scala spark application that loads parquet files from S3 and aggregates them:</p>
<pre><code>sparkSession.read.parquet(paths).as[MyRawEvent]
</code></pre>
<p>Our application runs perfectly on the happy-path: driver pod starts running, executors pods join the party, and when the application finishes both executors and driver terminate.</p>
<p>On the other hand, if something goes wrong, both driver + executors pods stay on <code>Running</code> state. For instance, an exception occurs (in driver) if one of the <code>paths</code> above doesn't exist:</p>
<pre><code>Exception in thread &quot;main&quot; org.apache.spark.sql.AnalysisException: Path does not exist: s3a://&lt;bucket-name&gt;/client-id=8765432/date=2021-08-06
     at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$4(DataSource.scala:803)
     at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$4$adapted(DataSource.scala:800)
     at org.apache.spark.util.ThreadUtils$.$anonfun$parmap$2(ThreadUtils.scala:372)
     at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
     at scala.util.Success.$anonfun$map$1(Try.scala:255)
     at scala.util.Success.map(Try.scala:213)
     at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
     at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
     at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
     at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
     at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
     at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
     at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
     at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
     at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
</code></pre>
<p>Interestingly, this exception doesn't prevent the executors to start right after and both driver&amp;executors pods are stuck forever, doing nothing.</p>
<p>We are not catching exceptions in our application and we expected that the driver and the executors will stop, instead of wasting redundant resources.</p>
<p>How can we crush the application so it won't stay in <code>Running</code> state forever?</p>

ItayB
<p>Well, that's was an easy one.</p>
<p>I had to catch all exceptions to ensure that spark context is being closed no matter what:</p>
<pre><code>  def main(args: Array[String]): Unit = {
    // some code
    implicit val sparkSession = SparkSession.builder().getOrCreate
    try {
      // application code with potential exceptions
    } catch {
      case exception: Exception =&gt;
        sparkSession.close()
        throw exception
    }

    sparkSession.close()
  }
</code></pre>
<p>That way all resources are freed and the driver pod changes its state to <code>Error</code> as excepted.</p>
<p><strong>EDIT</strong> - on in the Scala fashion:</p>
<pre><code>  def main(args: Array[String]): Unit = {
    // some code
    implicit val sparkSession = SparkSession.builder().getOrCreate
    Try {
      // application code with potential exceptions
    } match {
      case Success(_) =&gt; None
      case Failure(exception) =&gt;
        sparkSession.close()
        throw exception
    }

    sparkSession.close()
  }
</code></pre>
