title: Sparklyr: how to explode a list column into their own columns in Spark table?
body: <p>My question is similar with the one in <a href="https://stackoverflow.com/q/41810015/7913448">here</a>, but I'm having problems implementing the answer, and I cannot comment in that thread.</p>
  
  <p>So, I have a big CSV file that contains a nested data, which contains 2 columns separated by whitespace (say first column is Y, second column is X). Column X itself is also a comma-separated value.</p>
  
  <pre><code>21.66 2.643227,1.2698358,2.6338573,1.8812188,3.8708665,...
  35.15 3.422151,-0.59515584,2.4994135,-0.19701914,4.0771823,...
  15.22 2.8302398,1.9080592,-0.68780196,3.1878228,4.6600842,...
  ...
  </code></pre>
  
  <p>I want to read this CSV into 2 different Spark tables using <code>sparklyr</code>. </p>
  
  <p>So far this is what I've been doing:</p>
  
  <ol>
  <li><p>Use <code>spark_read_csv</code> to import all CSV contents into Spark data table</p>
  
  <p><code>df = spark_read_csv(sc, path = "path", name = "simData", delimiter = " ", header = "false", infer_schema = "false")</code></p>
  
  <p>The result is a Spark table named <code>simData</code> with 2 columns: <code>C0</code> and <code>C1</code></p></li>
  <li><p>Use <code>dplyr</code> to select first &amp; second column, and then register them as new tables named Y and X respectively</p>
  
  <p><code>simY &lt;- df %&gt;% select(C0) %&gt;% sdf_register("simY")</code></p>
  
  <p><code>simX &lt;- df %&gt;% select(C1) %&gt;% sdf_register("simX")</code></p></li>
  <li><p>Split the value in <code>simX</code> using <code>ft_regex_tokenizer</code> function, with regards the answer written in <a href="https://stackoverflow.com/q/41810015/7913448">here</a>. </p>
  
  <p><code>ft_regex_tokenizer(input_DF, input.col = "COL", output.col = "ResultCols", pattern = '\\###')</code></p></li>
  </ol>
  
  <p>But when I try to <code>head</code> it using <code>dplyr</code>:</p>
  
  <pre><code>Source:   query [6 x 1]
  Database: spark connection master=yarn-client app=sparklyr local=FALSE
  
          Result
          &lt;list&gt;
  1 &lt;list [789]&gt;
  2 &lt;list [789]&gt;
  3 &lt;list [789]&gt;
  4 &lt;list [789]&gt;
  5 &lt;list [789]&gt;
  6 &lt;list [789]&gt;
  </code></pre>
  
  <p>I want to turn this into a new Spark table and convert the type to double. Is there any way to do this? 
  I've considered to <code>collect</code> the data into R (using <code>dplyr</code>), convert to matrix, and then do <code>strsplit</code> for each row, but I think this is not a solution because the CSV size can go up to 40GB.</p>
  
  <p>EDIT: Spark version is 1.6.0</p>