title: Spark - word count using java
body: <p>I'm quite new to Spark and I would like to extract features (basically count of words) from a text file using the <code>Dataset</code> class. I have read the <a href="https://spark.apache.org/docs/latest/ml-features.html" rel="nofollow noreferrer">"Extracting, transforming and selecting features"</a> tutorial on Spark  but every example reported starts from a bag of words defined "on the fly". I have tried several times to generate the same kind of Dataset starting from a text file but I have always failed. Here is my code:</p>
  
  <pre class="lang-java prettyprint-override"><code>SparkSession spark = SparkSession
                .builder()
                .appName("Simple application")
                .config("spark.master", "local")
                .getOrCreate();
  
  Dataset&lt;String&gt; textFile = spark.read()
              .textFile("myFile.txt")
              .as(Encoders.STRING());
  
  Dataset&lt;Row&gt; words = textFile.flatMap(s -&gt; {
      return  Arrays.asList(s.toLowerCase().split("AG")).iterator();
      }, Encoders.STRING()).filter(s -&gt; !s.isEmpty()).toDF();
  
  Word2Vec word2Vec = new Word2Vec()
        .setInputCol("value")
        .setOutputCol("result")
        .setVectorSize(16)
        .setMinCount(0);
  
  Word2VecModel model = word2Vec.fit(words);
  Dataset&lt;Row&gt; result = model.transform(words);
  </code></pre>
  
  <p>I get this error message: <code>Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Column value must be of type equal to one of the following types: [ArrayType(StringType,true), ArrayType(StringType,false)] but was actually of type StringType.</code></p>
  
  <p>I think I have to convert each line into a Row using something like:</p>
  
  <pre class="lang-java prettyprint-override"><code>RowFactory.create(0.0, line)
  </code></pre>
  
  <p>but I cannot figure out how to do that.</p>
  
  <p>Basically, I was trying to train a classification system based on word counts of strings generated from a long sequence of characters. My text file contains one sequence per line so I need to split and count them for each row (the sub-strings are called k-mers and a general description can be found <a href="https://en.wikipedia.org/wiki/K-mer" rel="nofollow noreferrer">here</a>). Depending on the length of the k-mers I could have more than 4^32 different strings, so I was looking for a scalable machine learning algorithm like Spark.</p>