title: spark sql percentile over floating point column
body: <p>According to the <a href="https://spark.apache.org/docs/2.3.2/api/sql/#percentile" rel="nofollow noreferrer" title="spark sql 2.3.2">documentation</a> the percentile command should give the exact percentile for every numeric column. At least when my input is composed of floating points - this is not true. </p>
  
  <p>In <a href="https://support.treasuredata.com/hc/en-us/articles/360001457367-Hive-Built-in-Aggregate-Functions" rel="nofollow noreferrer">hive docs,</a> it says the percentile command only works for integers. I didn't understand the relation between spark sql and hive, but it seems spark doesn't just run hive - otherwise it wouldn't have changed it's docs. Also, spark's <code>percentile</code> has a different signature, allowing it to get a <code>frequency</code> parameter, which I also have no idea what purpose it serves.
  This is an example with unexpected output:</p>
  
  <pre><code>from pyspark.sql import functions as sf
  
  d = spark.createDataFrame([[35.138071000000004], [34.119932999999996], [34.487992]], ['a'])
  d.select(sf.expr('percentile(a, array(0.25,0.5,0.75,0.9,0.95)) AS res')).collect()
  
  Out[1]: [Row(res=[34.3039625, 34.487992, 34.8130315, 35.0080552, 35.0730631])]
  </code></pre>
  
  <p>If I switch <code>sf.expr</code> content to <code>percentile_approx</code> with a high accuracy, or use high <code>frequency</code> in the current method - I get a reasonable output.
  Could you explain what's happening?</p>
  
  <p>Also:</p>
  
  <ol>
  <li>Can you please explain/point me to some resource about the relation between spark sql and apache hive?</li>
  <li>Where is the code that spark sql commands run?</li>
  </ol>
  
  <p>Thanks</p>