title: reading and writing from hive tables with spark after aggregation
body: <p>We have a hive warehouse, and wanted to use spark for various tasks (mainly classification). At times write the results back as a hive table. For example, we wrote the following python function to find the total sum of original_table column two, grouped by original_table column one. The function works, but we are worried that it is inefficient, particularly the maps to convert to key-value pairs, and dictionary versions. Functions combiner, mergeValue, mergeCombiner are defined elsewhere, but work fine.</p>
  
  <pre><code>from pyspark import HiveContext
  
  rdd = HiveContext(sc).sql('from original_table select *')
  
  #convert to key-value pairs
  key_value_rdd = rdd.map(lambda x: (x[0], int(x[1])))
  
  #create rdd where rows are (key, (sum, count)
  combined = key_value_rdd.combineByKey(combiner, mergeValue, mergeCombiner)
  
  # creates rdd with dictionary values in order to create schemardd
  dict_rdd = combined.map(lambda x: {'k1': x[0], 'v1': x[1][0], 'v2': x[1][1]})
  
  # infer the schema
  schema_rdd = HiveContext(sc).inferSchema(dict_rdd)
  
  # save
  schema_rdd.saveAsTable('new_table_name')
  </code></pre>
  
  <p>Are there more efficient ways of doing the same thing?</p>