title: Merging lists in a single pySpark dataframe
body: <p>I am going through the pySpark 1.6.2 documentation in order to merge my data into a single dataframe.</p>
  
  <p>I have a list of 19 items (listname:sizes):</p>
  
  <pre><code>[9, 78, 13, 3, 57, 60, 66, 32, 24, 1, 2, 15, 2, 2, 76, 79, 100, 73, 4]
  </code></pre>
  
  <p>and a 2D list containing 19 not same length sub-lists (listname:data):</p>
  
  <pre><code>[[a,b,c],[d,e,f,g,h,i,j].......[x,y,z,a,f]]
  </code></pre>
  
  <p>I am trying to create a dataframe that looks like this:</p>
  
  <pre><code>        name                   size
  0     [a,b,c]                  9
  1     [d,e,f,g,h,i,j]          78
  2     ........                 ...
  .     ........                 ...
  .     ........                 ...
  18    [x,y,z,a,f]              4
  </code></pre>
  
  <p>But I can't figure out a way to do that. </p>
  
  <p>I have already iterated through the list and I could append the two columns after each iteration.
  But I am finding it hard to find a way to create a Dataframe and fill it step by step.</p>
  
  <p>This is my code:</p>
  
  <pre><code>schema = StructType([StructField("name", StringType(), True), StructField("size", IntegerType(), True)])
  dataframe = sqlContext.createDataFrame([],schema)
  
  for i in range(len(data)):
      sizes.append(len(data[i]))
      t = sqlContext.DataFrame([[data[i], sizes[i]]],
                           columns=['name', 'size'])
      dataframe = dataframe.append(t, ignore_index=True)
  
  dataframe.show()
  </code></pre>
  
  <p>but it returns me this:</p>
  
  <pre><code>+----+----+
  |name|size|
  +----+----+
  +----+----+
  </code></pre>