title: Databricks SCALA UDF cannot load class when registering function
body: <p>I have followed <a href="https://databricks.com/notebooks/enforcing-column-level-encryption.html" rel="nofollow noreferrer">this guide</a> and <a href="https://stackoverflow.com/questions/56964275/creating-user-defined-not-temporary-function-in-spark-sql-for-azure-databricks">this question</a> trying to implement a decryption function to use in a SQL view.</p>
  <p>I have compiled this scala code in the example to a jar file and uploaded to the Databricks File System (DBFS):</p>
  <pre class="lang-scala prettyprint-override"><code>import com.macasaet.fernet.{Key, StringValidator, Token};
  import org.apache.hadoop.hive.ql.exec.UDF;
  import java.time.{Duration, Instant};
  
  class Validator extends StringValidator {
      override def getTimeToLive() : java.time.temporal.TemporalAmount = {
        Duration.ofSeconds(Instant.MAX.getEpochSecond());
      }
    }
  
  class udfDecrypt extends UDF {
  
      def evaluate(inputVal: String, sparkKey : String): String = {
  
        if( inputVal != null &amp;&amp; inputVal!=&quot;&quot; ) {
          val keys: Key = new Key(sparkKey)
          val token = Token.fromString(inputVal)
          val validator = new Validator() {}
          val payload = token.validateAndDecrypt(keys, validator)
          payload
        } else return inputVal
      }
    }
  </code></pre>
  <p>I can declare the function as demonstrated:</p>
  <pre class="lang-sql prettyprint-override"><code>%sql
  CREATE OR REPLACE FUNCTION default.udfDecrypt AS 'com.nm.udf.udfDecrypt'
  USING jar 'dbfs:/FileStore/jars/decryptUDF.jar';
  </code></pre>
  <p>But if I try to call it an error is thrown:</p>
  <pre class="lang-sql prettyprint-override"><code>%sql
  SELECT default.udfDecrypt(field, '{key}') FROM default.encrypted_test;
  </code></pre>
  <p><code>Error in SQL statement: AnalysisException: Can not load class 'com.nm.udf.udfDecrypt' when registering the function 'default.udfDecrypt', please make sure it is on the classpath; line 1 pos 7</code></p>
  <p>I have noticed that the function can be declared using any jar file path (even one that doesn't exist) and it will still return 'OK'.</p>
  <p>I am using Databricks for Azure.</p>