title: saving a list of rows to a Hive table in pyspark
body: <p>I have a pyspark app.  I copied a hive table to my hdfs directory, &amp; in python I <code>sqlContext.sql</code> a query on this table. Now this variable is a dataframe I call <code>rows</code>.  I need to randomly shuffle the <code>rows</code>, so I had to convert them to a list of rows <code>rows_list = rows.collect()</code>.  So then I <code>shuffle(rows_list)</code> which shuffles the lists in place.   I take the amount of random rows I need <code>x</code>:</p>
  
  <p><code>for r in range(x):
          allrows2add.append(rows_list[r])
  </code>
  Now I want to save allrows2add as a hive table OR append an existing hive table (whichever is easier to do).  The problem is that I can not do this:</p>
  
  <p><code>all_df = sc.parallelize(allrows2add).toDF()</code> Cant do this, schema can't be inferred
  <code>ValueError: Some of types cannot be determined by the first 100 rows, please try again with sampling</code></p>
  
  <p>without putting in the whole schema.  The schema of <code>rows</code> has 117 columns, so I don't want to type them out.  Is there a way to extract the schema of <code>rows</code> to help me make allrows2add a dataframe or somehow save as a hive table?
  I can do 
  <code>rows.printSchema()</code> but not sure how to get it into a schema format as a variable to pass <code>toDF()</code> without having to parse all of that text</p>
  
  <p>Thanks</p>
  
  <p>Adding for loop info</p>
  
  <pre><code>#Table is a List of Rows from small Hive table I loaded using
  #query = "SELECT * FROM Table"
  #Table = sqlContext.sql(query).collect()
  
  for i in range(len(Table)):
  
      rows = sqlContext.sql(qry)
      val1 = Table[i][0]
      val2 = Table[i][1]
      count = Table[i][2]
      x = 100 - count
  
  #hivetemp is a table that I copied from Hive to my hfs using:
  #create external table IF NOT EXISTS hive temp LIKE hivetableIwant2copy LOCATION "/user/name/hiveBackup";
  #INSERT OVERWRITE TABLE hivetemp SELECT * FROM hivetableIwant2copy;
  
      query = "SELECT * FROM hivetemp WHERE col1&lt;&gt;\""+val1+"\" AND col2 ==\""+val2+"\" ORDER BY RAND() LIMIT "+str(x)
  
      rows = sqlContext.sql(query)
      rows = rows.withColumn("col4", lit(10))
      rows = rows.withColumn("col5", lit(some_string))
  #writing to parquet is heck slow AND I can't work with pandas due to the library not installed on the server
      rows.saveAsParquetFile("rows"+str(i)+".parquet")
  #tried this before and heck slow also
      #rows_list = rows.collect()
      #shuffle(rows_list)
  </code></pre>