title: Select specific columns in a PySpark dataframe to improve performance
body: <p>Working with Spark dataframes imported from Hive, sometimes I end up with several columns that I don't need. Supposing that I don't want to filter them with </p>
  
  <pre><code>df = SqlContext.sql('select cols from mytable')
  </code></pre>
  
  <p>and I'm importing the entire table with </p>
  
  <pre><code>df = SqlContext.table(mytable)
  </code></pre>
  
  <p>does a <code>select</code> and subsequent <code>cache</code> improves performance/decrease memory usage, like</p>
  
  <pre><code>df = df.select('col_1', 'col_2', 'col_3')
  df.cache()
  df.count()
  </code></pre>
  
  <p>or is just waste of time? I will do lots of operations and data manipulations on <code>df</code>, like <code>avg</code>, <code>withColumn</code>, etc.</p>