title: How to release a dataframe in spark?
body: <p>I am using spark do some calculation. every 5 minutes, I got a new data frame. I put it into a dict called dict_1_hour like this</p>
  
  <pre><code>dict_1_hour[timestamp] = dataframe
  </code></pre>
  
  <p>New data frame comes in to the dict and old dataframe pop out from the dict. Only 12 dataframe are kept in it, that is the data for the recent 1 hour.</p>
  
  <p>So my question is how should I release those dataframe to make sure there is no memory leak?</p>
  
  <p>One API for dataframe seems can do it.(I do not know what the parameter for)</p>
  
  <pre><code>unpersist(blocking=True)
  Marks the DataFrame as non-persistent, and remove all blocks for it from memory and disk.
  </code></pre>
  
  <p>The other way I think is just pop the dataframe from the dict.</p>
  
  <pre><code>dict_1_hour.pop(timestamp)
  </code></pre>
  
  <p>Python should auto release the unused variable. But I do not know if it is appropriate here. I worried that spark may keep the dataframe if you do not release it explicit</p>
  
  <p>So please advice on which way I should use please</p>