title: Can I append to a compressed stream with pandas?
body: <p>I know that by passing the <code>compression='gzip'</code> argument to <code>pd.to_csv()</code> I can save a DataFrame into a compressed CSV file.</p>
  <pre><code>my_df.to_csv('my_file_name.csv', compression='gzip')
  </code></pre>
  <p>I also know that if I want to <em>append</em> a DataFrame to the end of an existing CSV file I can use <code>mode='a'</code>, like so</p>
  <pre><code>my_df.to_csv('my_file_name.csv', mode='a', index=False)
  </code></pre>
  <p>But what if I want to append a DataFrame to the end of a compressed CSV file? Is that even possible? I tried to do so with</p>
  <pre><code>my_df.to_csv('my_file_name.csv', mode='a', index=False, compression='gzip')
  </code></pre>
  <p>But the resulting CSV was not compressed, albeit in fine condition.</p>
  <hr />
  <p>This question is motivated by my processing of a large CSV file with Pandas. I need to produce compressed CSV output, and am processing the CSV file in chunks into a DataFrame so that I don't run into a MemoryError. Hence, the most seemingly logical thing for me to do is to append each output DataFrame chunk together into one compressed zip file.</p>
  <p>I am using Python 3.4 and Pandas 0.16.1.</p>