title: Hive query vs python list processing performance
body: <p>Scenario:
  I have a python application which connects to hive database and extracts data, creates sub tables for processing data,drops temp tables etc.. 90% of the processing is done using hive queries which is essentially generated from python code and sends to hive server to execute.</p>
  
  <ul>
  <li><p>There is a minimum time spent on sending each query to hive server,
  and returning the status or data. I understand there are unavoidable
  scenarios where the data is in  hive.</p></li>
  <li><p>Then there is processing time of the hive query.</p></li>
  </ul>
  
  <p>There are scenarios where It consumes lot of time to process each query when there are lot of queries to execute in a loop.</p>
  
  <p>Question :
  If we assume the data is like some 50k rows. Would it be much faster to fetch 50k rows to python itself(ofcourse the data transfer time is overhead here)- like into a list of tuples- and mimic the join/filter operations hive does and avoid the 10-50 queries run against hive?</p>
  
  <p>Would this be a better approach? Or should i look for hive query optimization techniques. I prefer hive queries as it is convenient to use.</p>