title: Sparklyr handing categorical variables
body: <h1>Sparklyr handling categorical variables</h1>
  
  <p>I came from R background and I am used to categorical variables being handled in the backend (as factor). With Sparklyr it is quite confusing using <code>string_indexer</code> or <code>onehotencoder</code>. </p>
  
  <p>For example, I have a number of variables which has been encoded as numerical variables in the original dataset but they are actually categorical. I want to use them as categorical variables but am not sure I am doing it correctly. </p>
  
  <pre><code>library(sparklyr)
  library(dplyr)
  sessionInfo()
  sc &lt;- spark_connect(master = "local", version = spark_version)
  spark_version(sc)
  set.seed(1)    
  exampleDF &lt;- data.frame (ID = 1:10, Resp = sample(c(100:205), 10, replace = TRUE), 
                       Numb = sample(1:10, 10))
  
  example &lt;- copy_to(sc, exampleDF) 
  pred &lt;- example %&gt;% mutate(Resp = as.character(Resp)) %&gt;%
                  sdf_mutate(Resp_cat = ft_string_indexer(Resp)) %&gt;%
                  ml_decision_tree(response = "Resp_cat", features = "Numb") %&gt;%
                  sdf_predict()
  pred
  </code></pre>
  
  <p>The prediction from the model is not categorical. See below. Does it mean I also have to convert back from prediction to Resp_cat and then to Resp?</p>
  
  <pre><code>R version 3.4.0 (2017-04-21)
  Platform: x86_64-redhat-linux-gnu (64-bit)
  Running under: CentOS Linux 7 (Core)
  
  spark_version(sc)
  [1] ‘2.1.1.2.6.1.0’
  
  Source:   table&lt;sparklyr_tmp_74e340c5607c&gt; [?? x 6]
  Database: spark_connection
        ID  Numb  Resp Resp_cat id74e35c6b2dbb prediction
       &lt;int&gt; &lt;int&gt; &lt;chr&gt;    &lt;dbl&gt;          &lt;dbl&gt;      &lt;dbl&gt;
   1     1    10   150        8              0   8.000000
   2     2     3   191        4              1   4.000000
   3     3     4   146        9              2   9.000000
   4     4     9   125        5              3   5.000000
   5     5     8   107        2              4   2.000000
   6     6     2   110        1              5   1.000000
   7     7     5   133        3              6   5.333333
   8     8     7   154        6              7   5.333333
   9     9     1   170        0              8   0.000000
  10    10     6   143        7              9   5.333333
  </code></pre>