title: Spark MLLib TFIDF implementation for LogisticRegression
body: <p>I try to use the new TFIDF algorithem that spark 1.1.0 offers. I'm writing my job for MLLib in Java but I can't figure out how to get the TFIDF implementation working. For some reason <a href="http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/feature/IDFModel.html" rel="nofollow">IDFModel</a> only accepts a <a href="http://spark.apache.org/docs/latest/api/java/org/apache/spark/api/java/JavaRDD.html" rel="nofollow">JavaRDD</a> as input for the method <a href="http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/feature/IDFModel.html#transform%28org.apache.spark.api.java.JavaRDD%29" rel="nofollow">transform</a> and not simple Vector. <strong>How can I use the given classes to model a TFIDF vector for my LabledPoints?</strong></p>
  
  <p><em>Note: The document lines are in the format [Label; Text]</em></p>
  
  <hr>
  
  <p>Here my code so far:</p>
  
  <pre><code>        // 1.) Load the documents
          JavaRDD&lt;String&gt; data = sc.textFile("/home/johnny/data.data.new"); 
  
          // 2.) Hash all documents
          HashingTF tf = new HashingTF();
          JavaRDD&lt;Tuple2&lt;Double, Vector&gt;&gt; tupleData = data.map(new Function&lt;String, Tuple2&lt;Double, Vector&gt;&gt;() {
              @Override
              public Tuple2&lt;Double, Vector&gt; call(String v1) throws Exception {
                  String[] data = v1.split(";");
                  List&lt;String&gt; myList = Arrays.asList(data[1].split(" "));
                  return new Tuple2&lt;Double, Vector&gt;(Double.parseDouble(data[0]), tf.transform(myList));
              }
          });
  
          tupleData.cache();
  
          // 3.) Create a flat RDD with all vectors
          JavaRDD&lt;Vector&gt; hashedData = tupleData.map(new Function&lt;Tuple2&lt;Double,Vector&gt;, Vector&gt;() {
              @Override
              public Vector call(Tuple2&lt;Double, Vector&gt; v1) throws Exception {
                  return v1._2;
              }
          });
  
          // 4.) Create a IDFModel out of our flat vector RDD
          IDFModel idfModel = new IDF().fit(hashedData);
  
          // 5.) Create Labledpoint RDD with TFIDF
          ???
  </code></pre>
  
  <p><strong>Solution</strong>  <em>from Sean Owen</em>:</p>
  
  <pre><code>        // 1.) Load the documents
          JavaRDD&lt;String&gt; data = sc.textFile("/home/johnny/data.data.new"); 
  
          // 2.) Hash all documents
          HashingTF tf = new HashingTF();
          JavaRDD&lt;LabeledPoint&gt; tupleData = data.map(v1 -&gt; {
                  String[] datas = v1.split(";");
                  List&lt;String&gt; myList = Arrays.asList(datas[1].split(" "));
                  return new LabeledPoint(Double.parseDouble(datas[0]), tf.transform(myList));
          }); 
          // 3.) Create a flat RDD with all vectors
          JavaRDD&lt;Vector&gt; hashedData = tupleData.map(label -&gt; label.features());
          // 4.) Create a IDFModel out of our flat vector RDD
          IDFModel idfModel = new IDF().fit(hashedData);
          // 5.) Create tfidf RDD
          JavaRDD&lt;Vector&gt; idf = idfModel.transform(hashedData);
          // 6.) Create Labledpoint RDD
          JavaRDD&lt;LabeledPoint&gt; idfTransformed = idf.zip(tupleData).map(t -&gt; {
              return new LabeledPoint(t._2.label(), t._1);
          });
  </code></pre>