title: spark (java) - Too many open files
body: <p>I am trying to run a batch job in spark2 which takes a huge list as input and iterates on the list to perform the processing. The program executes fine for around 8 000 records ​of the list and then breaks giving exception:</p>
  
  <pre><code>WARN Lost task 0.0 in stage 421079.0 (TID 996338, acusnldlenhww4.cloudapp.net, executor 1): java.io.FileNotFoundException: /data/1/hadoop/yarn/local/usercache/A2159537-MSP01/appcache/application_1497532405817_0072/blockmgr-73dc563c-8ea5-4f2d-adfe-6c60cf3e3968/0d/shuffle_145960_0_0.index.cfb6d5ea-8c7b-41a1-acc3-2c840e7f8998 (Too many open files)
          at java.io.FileOutputStream.open0(Native Method)
          at java.io.FileOutputStream.open(FileOutputStream.java:270)
          at java.io.FileOutputStream.&lt;init&gt;(FileOutputStream.java:213)
          at java.io.FileOutputStream.&lt;init&gt;(FileOutputStream.java:162)
          at org.apache.spark.shuffle.IndexShuffleBlockResolver.writeIndexFileAndCommit(IndexShuffleBlockResolver.scala:144)
          at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:128)
          at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
          at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
          at org.apache.spark.scheduler.Task.run(Task.scala:99)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
       (org.apache.spark.scheduler.TaskSetManager)
  </code></pre>
  
  <p>neo4j database is used as input. I am reading 300k nodes from neo4j as input and do a for loop on the input rdd.</p>
  
  <p>Tried setting <code>spark.shuffle.consolidateFiles</code> to <code>true</code> in SparkConf. But that didn't work.</p>