title: Spark SPEL Expressions UDF
body: <p>I'm trying to use SPEL expressions along with a Spark udf (scala). SPEL expressions can be simple math operations or custom functions along with java.lang.Math class. When i run it locally everything works find, but when i build a fat jar and deploy it on databricks, i'm getting the following exception</p>
  
  <pre><code>SparkException: Job aborted due to stage failure: Task 0 in stage 159.0 failed 4 times, most recent failure: Lost task 0.3 in stage 159.0 (TID 1290, 172.16.9.7, executor 2): java.lang.NoSuchMethodError: org.springframework.core.convert.support.DefaultConversionService.getSharedInstance()Lorg/springframework/core/convert/ConversionService;
  </code></pre>
  
  <p>I get this exception when the SPEL contains a custom function or java.lang.Math class, but it works then there are simple Math operations like addition, sub, mul etc.</p>
  
  <p>Searching online indicates that this is caused due to version incompatibilities in spring jars, but 1. im only using spring-expression and 2. i dont see any version incompatibilities.</p>
  
  <p>This is my UDF:</p>
  
  <pre><code> def calculateSpelExpression: (Seq[Double], String, mutable.WrappedArray[String]) =&gt; Double = (values: Seq[Double], expression: String, variableNames: mutable.WrappedArray[String]) =&gt; {
      val context = new StandardEvaluationContext()
      val parser = new SpelExpressionParser()
  
      variableNames.zipWithIndex.foreach { iter =&gt;
        val name = iter._1.tail // use .tail in order to remove the '# ' symbol
        val value = values(iter._2)
        context.setVariable(name, value)
      }
      val value = parser.parseExpression(expression).getValue(context)
      if (value.isInstanceOf[Int]) value.asInstanceOf[Int].toDouble
      else value.asInstanceOf[Double]
    }
  </code></pre>
  
  <p>and my <code>pom.xml</code>:</p>
  
  <pre><code>
      &lt;dependencies&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;com.typesafe&lt;/groupId&gt;
              &lt;artifactId&gt;config&lt;/artifactId&gt;
              &lt;version&gt;1.4.0&lt;/version&gt;
          &lt;/dependency&gt;
  
          &lt;dependency&gt;
              &lt;groupId&gt;org.springframework&lt;/groupId&gt;
              &lt;artifactId&gt;spring-expression&lt;/artifactId&gt;
              &lt;version&gt;5.2.0.RELEASE&lt;/version&gt;
          &lt;/dependency&gt;
  
          &lt;dependency&gt;
              &lt;groupId&gt;org.scala-lang&lt;/groupId&gt;
              &lt;artifactId&gt;scala-library&lt;/artifactId&gt;
              &lt;version&gt;${scala.version}&lt;/version&gt;
          &lt;/dependency&gt;
  
          &lt;dependency&gt;
              &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
              &lt;artifactId&gt;spark-sql_2.11&lt;/artifactId&gt;
              &lt;version&gt;${spark.version}&lt;/version&gt;
          &lt;/dependency&gt;
  
          &lt;dependency&gt;
              &lt;groupId&gt;org.scalatest&lt;/groupId&gt;
              &lt;artifactId&gt;scalatest_${scala.compat.version}&lt;/artifactId&gt;
              &lt;version&gt;3.0.8&lt;/version&gt;
              &lt;scope&gt;test&lt;/scope&gt;
          &lt;/dependency&gt;
  
   &lt;dependency&gt;
              &lt;groupId&gt;org.apache.commons&lt;/groupId&gt;
              &lt;artifactId&gt;commons-math3&lt;/artifactId&gt;
              &lt;version&gt;3.2&lt;/version&gt;
          &lt;/dependency&gt;
  
          &lt;dependency&gt;
              &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
              &lt;artifactId&gt;spark-mllib_2.11&lt;/artifactId&gt;
              &lt;version&gt;${spark.version}&lt;/version&gt;
          &lt;/dependency&gt;
  
  
          &lt;dependency&gt;
              &lt;groupId&gt;io.delta&lt;/groupId&gt;
              &lt;artifactId&gt;delta-core_2.11&lt;/artifactId&gt;
              &lt;version&gt;0.4.0&lt;/version&gt;
  &lt;!--            &lt;scope&gt;provided&lt;/scope&gt;--&gt;
          &lt;/dependency&gt;
  
          &lt;dependency&gt;
              &lt;groupId&gt;org.postgresql&lt;/groupId&gt;
              &lt;artifactId&gt;postgresql&lt;/artifactId&gt;
              &lt;version&gt;42.2.10&lt;/version&gt;
          &lt;/dependency&gt;
  
          &lt;dependency&gt;
              &lt;groupId&gt;com.datastax.spark&lt;/groupId&gt;
              &lt;artifactId&gt;spark-cassandra-connector_2.11&lt;/artifactId&gt;
              &lt;version&gt;${spark.version}&lt;/version&gt;
              &lt;exclusions&gt;
                  &lt;exclusion&gt;
                      &lt;groupId&gt;io.netty&lt;/groupId&gt;
                      &lt;artifactId&gt;netty-all&lt;/artifactId&gt;
                  &lt;/exclusion&gt;
              &lt;/exclusions&gt;
          &lt;/dependency&gt;
      &lt;/dependencies&gt;
  </code></pre>