title: Spark Streaming UpdateStateByKey
body: <p>I am running a spark streaming 24X7 and using updateStateByKey function to save the computed historical data like in the case of NetworkWordCount Example..</p>
  
  <p>I am tried to stream a file with 3lac records with 1 sec sleep for every 1500 records.
  I am using 3 workers</p>
  
  <ol>
  <li>Over a period updateStateByKey is growing, then the program throws the following exception</li>
  </ol>
  
  <p>ERROR Executor: Exception in task ID 1635
  java.lang.ArrayIndexOutOfBoundsException: 3</p>
  
  <pre><code>14/10/23 21:20:43 ERROR TaskSetManager: Task 29170.0:2 failed 1 times; aborting job
  14/10/23 21:20:43 ERROR DiskBlockManager: Exception while deleting local spark dir: /var/folders/3j/9hjkw0890sx_qg9yvzlvg64cf5626b/T/spark-local-20141023204346-b232
  java.io.IOException: Failed to delete: /var/folders/3j/9hjkw0890sx_qg9yvzlvg64cf5626b/T/spark-local-20141023204346-b232/24
  
  14/10/23 21:20:43 ERROR Executor: Exception in task ID 8037
  java.io.FileNotFoundException: /var/folders/3j/9hjkw0890sx_qg9yvzlvg64cf5626b/T/spark-local-20141023204346-b232/22/shuffle_81_0_1 (No such file or directory)
      at java.io.FileOutputStream.open(Native Method)
  </code></pre>
  
  <p>How to handle this?
  I guess updateStateByKey should be periodically reset as its growing in a rapid rate, please share some example on when and how to reset the updateStateByKey.. or i there any other problem? shed some light.</p>
  
  <p>Any help is much appreciated. Thanks for your time </p>