title: Find Interval Numbers with Quantile function on a Dataframe with Spark using Scala
body: <p>I'm trying to port a code from R to Scala to perform Customer Analysis. I have already computed Recency, Frequency and Monetary factors on Spark into a DataFrame.</p>
  
  <p>Here is the schema of the Dataframe :</p>
  
  <pre><code>df.printSchema 
  root
   |-- customerId: integer (nullable = false)
   |-- recency: long (nullable = false)
   |-- frequency: long (nullable = false)
   |-- monetary: double (nullable = false)
  </code></pre>
  
  <p>And here is a data sample as well :</p>
  
  <pre><code>df.order($"customerId").show 
  
  +----------+-------+---------+------------------+
  |customerId|recency|frequency|          monetary|
  +----------+-------+---------+------------------+
  |         1|    297|      114|            733.27|
  |         2|    564|       11|            867.66|
  |         3|   1304|        1|             35.89|
  |         4|    287|       25|            153.08|
  |         6|    290|       94|           316.772|
  |         8|   1186|        3|            440.21|
  |        11|    561|        5|            489.70|
  |        14|    333|       57|            123.94|
  </code></pre>
  
  <p>I'm trying to find the intervals for on a quantile vector for each column given a probability segment. </p>
  
  <p>In other words, given a probability vector of non-decreasing breakpoints,
  in my case it will be the quantile vector, find the interval containing each element of x; </p>
  
  <p>i.e. (pseudo-code), </p>
  
  <pre><code>if i &lt;- findInterval(x,v), 
  for each index j in x 
      v[i[j]] ≤ x[j] &lt; v[i[j] + 1] where v[0] := - Inf, v[N+1] := + Inf, and N &lt;- length(v). 
  </code></pre>
  
  <p>In R, this translates to the following code :</p>
  
  <pre><code>probSegment &lt;- c(0.0, 0.25, 0.50, 0.75, 1.0)
  
  RFM_table$Rsegment &lt;- findInterval(RFM_table$Recency, quantile(RFM_table$Recency, probSegment)) 
  RFM_table$Fsegment &lt;- findInterval(RFM_table$Frequency, quantile(RFM_table$Frequency, probSegment)) 
  RFM_table$Msegment &lt;- findInterval(RFM_table$Monetary, quantile(RFM_table$Monetary, probSegment))
  </code></pre>
  
  <p>I'm kind of stuck with the quantile function thought.</p>
  
  <p>In an earlier <a href="https://stackoverflow.com/questions/31432843/how-to-find-median-using-pyspark/31437177?noredirect=1#comment53486994_31437177">discussion</a> with @zero323, he suggest that I used the <code>percentRank</code> window function which can be used as a shortcut. I'm not sure that I can apply the percentRank function in this case.</p>
  
  <p>How can I apply a quantile function on a Dataframe column with Scala Spark? If this is not possible, can I use the percentRank function instead?</p>
  
  <p>Thanks.</p>