title: Too many open files during Cassandra nodetool repair
body: <p>I made a nodetool repair command on one node. This node went down, and the log files showed the following error message:</p>
  
  <pre><code>INFO  [STREAM-IN-/192.168.2.100] 2015-02-13 21:36:23,077 StreamResultFuture.java:180 - [Stream #8fb54551-b3bd-11e4-9620-4b92877f0505] Session with /192.168.2.100 is complete
  INFO  [STREAM-IN-/192.168.2.100] 2015-02-13 21:36:23,078 StreamResultFuture.java:212 - [Stream #8fb54551-b3bd-11e4-9620-4b92877f0505] All sessions completed
  INFO  [STREAM-IN-/192.168.2.100] 2015-02-13 21:36:23,078 StreamingRepairTask.java:96 - [repair #508bd650-b3bd-11e4-9620-4b92877f0505] streaming task succeed, returning response to node4/192.168.2.104
  INFO  [AntiEntropyStage:1] 2015-02-13 21:38:52,795 RepairSession.java:237 - [repair #508bd650-b3bd-11e4-9620-4b92877f0505] repcode is fully synced
  INFO  [AntiEntropySessions:27] 2015-02-13 21:38:52,795 RepairSession.java:299 - [repair #508bd650-b3bd-11e4-9620-4b92877f0505] session completed successfully
  INFO  [AntiEntropySessions:27] 2015-02-13 21:38:52,795 RepairSession.java:260 - [repair #03858e40-b3be-11e4-9620-4b92877f0505] new session: will sync node4/192.168.2.104, /192.168.2.100, /192.168.2.101 on range (8805399388216156805,8848902871518111273] for data.[repcode]
  INFO  [AntiEntropySessions:27] 2015-02-13 21:38:52,795 RepairJob.java:145 - [repair #03858e40-b3be-11e4-9620-4b92877f0505] requesting merkle trees for repcode (to [/192.168.2.100, /192.168.2.101, node4/192.168.2.104])
  WARN  [StreamReceiveTask:74] 2015-02-13 21:41:58,544 CLibrary.java:231 - open(/user/jlor/apache-cassandra/data/data/data/repcode-398f26f0b11511e49faf195596ed1fd9, O_RDONLY) failed, errno (23).
  WARN  [STREAM-IN-/192.168.2.101] 2015-02-13 21:41:58,672 CLibrary.java:231 - open(/user/jlor/apache-cassandra/data/data/data/repcode-398f26f0b11511e49faf195596ed1fd9, O_RDONLY) failed, errno (23).
  WARN  [STREAM-IN-/192.168.2.101] 2015-02-13 21:41:58,871 CLibrary.java:231 - open(/user/jlor/apache-cassandra/data/data/data/repcode-398f26f0b11511e49faf195596ed1fd9, O_RDONLY) failed, errno (23).
  ERROR [StreamReceiveTask:74] 2015-02-13 21:41:58,986 CassandraDaemon.java:153 - Exception in thread Thread[StreamReceiveTask:74,5,main]
  org.apache.cassandra.io.FSWriteError: java.io.FileNotFoundException: /user/jlor/apache-cassandra/data/data/data/repcode-398f26f0b11511e49faf195596ed1fd9/data-repcode-tmp-ka-245139-TOC.txt (Too many open files in system)
          at org.apache.cassandra.io.sstable.SSTable.appendTOC(SSTable.java:282) ~[apache-cassandra-2.1.2.jar:2.1.2]
          at org.apache.cassandra.io.sstable.SSTableWriter.close(SSTableWriter.java:483) ~[apache-cassandra-2.1.2.jar:2.1.2]
          at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:434) ~[apache-cassandra-2.1.2.jar:2.1.2]
          at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:429) ~[apache-cassandra-2.1.2.jar:2.1.2]
          at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:424) ~[apache-cassandra-2.1.2.jar:2.1.2]
          at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:120) ~[apache-cassandra-2.1.2.jar:2.1.2]
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_31]
          at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_31]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_31]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_31]
          at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]
  Caused by: java.io.FileNotFoundException: /usr/jlo/apache-cassandra/data/data/data/repcode-398f26f0b11511e49faf195596ed1fd9/data-repcode-tmp-ka-245139-TOC.txt (Too many open files in system)
          at java.io.FileOutputStream.open(Native Method) ~[na:1.8.0_31]
          at java.io.FileOutputStream.&lt;init&gt;(FileOutputStream.java:213) ~[na:1.8.0_31]
          at java.io.FileWriter.&lt;init&gt;(FileWriter.java:107) ~[na:1.8.0_31]
          at org.apache.cassandra.io.sstable.SSTable.appendTOC(SSTable.java:276) ~[apache-cassandra-2.1.2.jar:2.1.2]
          ... 10 common frames omitted
  </code></pre>
  
  <p>We have a small cluster with 5 nodes: node0-node4. 
  I have one table with about 3.4 billion rows, with replica 3. 
  Here is the table description:</p>
  
  <pre><code>CREATE TABLE data.repcode (
  rep int,
  type text,
  code text,
  yyyymm int,
  trd int,
  eq map&lt;text, bigint&gt;,
  iq map&lt;text, bigint&gt;,
  PRIMARY KEY ((rep, type, pcode), yyyymm, trd))
  WITH CLUSTERING ORDER BY (yyyymm ASC, co_trd ASC, md5 ASC)
  AND bloom_filter_fp_chance = 0.1
  AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
  AND comment = ''
  AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy', 'max_threshold': '32'}
  AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
  AND dclocal_read_repair_chance = 0.1
  AND default_time_to_live = 0
  AND gc_grace_seconds = 864000
  AND max_index_interval = 2048
  AND memtable_flush_period_in_ms = 0
  AND min_index_interval = 128
  AND read_repair_chance = 0.0
  AND speculative_retry = '99.0PERCENTILE';
  </code></pre>
  
  <p>I'm using Cassandra 2.1.2. 
  I have set the limit of max open files for all my nodes to 200'000.</p>
  
  <p>Before I issued the nodetool repair command I had counted the file count in my data directories. Here is the count on each of my nodes before the crash:</p>
  
  <pre><code>node0: 27'099 
  node1: 27'187 
  node2: 36'131 
  node3: 26'635 
  node4: 26'371 
  </code></pre>
  
  <p>Now after the crash:</p>
  
  <pre><code>node0:   946'555 
  node1:   973'531 
  node2:   844'211 
  node3: 1'024'147 
  node4: 1'971'772 
  </code></pre>
  
  <p>Is it normal that the amount of file in one unix directory raise to such extend?
  What can I do to avoid this?
  How can I avoid this problem in the future? Should I increase the amount of open files? This seems already very big to me.
  Is my cluster too small for this amount of records?
  Should I use another compaction strategy?</p>
  
  <p>Thanks for your help.</p>