title: How big the spark stream window could be?
body: <p>I have some data flows need to be calculated. I am thinking about use spark stream to do this job. But there is one thing I am not sure and feel worry about.</p>
  
  <p>My requirements is like :</p>
  
  <p>Data comes in as CSV files every 5 minutes. I need report on data of recent 5 minutes, 1 hour and 1 day. So If I setup a spark stream to do this calculation. I need a interval as 5 minutes. Also I need to setup two window 1 hour and 1 day.</p>
  
  <p>Every 5 minutes there will be 1GB data comes in. So the one hour window will calculate 12GB (60/5) data and the one day window will calculate 288GB(24*60/5)  data.</p>
  
  <p>I do not have much experience on spark. So this worries me. </p>
  
  <ol>
  <li><p>Can spark handle such big window ?</p></li>
  <li><p>How much RAM do I need to calculation those 288 GB data?  More than 288 GB RAM? (I know this may depend on my disk I/O, CPU and the calculation pattern. But I just want some estimated answer based on experience)</p></li>
  <li><p>If calculation on one day / one hour data is too expensive in stream. Do you have any better suggestion?</p></li>
  </ol>