title: Hash function in spark
body: <p>I'm trying to add a column to a dataframe, which will contain <strong>hash of another column</strong>.</p>
  
  <p>I've found this piece of documentation:
  <a href="https://spark.apache.org/docs/2.3.0/api/sql/index.html#hash" rel="noreferrer">https://spark.apache.org/docs/2.3.0/api/sql/index.html#hash</a><br>
  And tried this:  </p>
  
  <pre><code>import org.apache.spark.sql.functions._
  val df = spark.read.parquet(...)
  val withHashedColumn = df.withColumn("hashed", hash($"my_column"))
  </code></pre>
  
  <p>But what is the hash function used by that <code>hash()</code>? Is that <code>murmur</code>, <code>sha</code>, <code>md5</code>, something else?  </p>
  
  <p>The value I get in this column is integer, thus range of values here is probably <code>[-2^(31) ... +2^(31-1)]</code>.<br>
  Can I get a long value here? Can I get a string hash instead?<br>
  How can I specify a concrete hashing algorithm for that?<br>
  Can I use a custom hash function?</p>