title: Elasticsearch - Too many open files after updating file limit
body: <p>I'm running into this error even after I've updated the file limit per node with my ES cluster. My set up is a 6-node cluster with around 630 million documents indexed.</p>
  
  <p>This error happened near the end of my restarting of the cluster:</p>
  
  <pre><code>    [2013-11-11 00:00:21,087][WARN ][index.engine.robin       ] [rap-es-3] [zt_2012][11] failed to read latest segment infos on flush
      java.io.FileNotFoundException: /data/elasticsearch/data/Odine/nodes/0/indices/zt_2012/11/index/segments_e (Too many open files)
      at java.io.RandomAccessFile.open(Native Method)
      at java.io.RandomAccessFile.&lt;init&gt;(Unknown Source)
      at org.apache.lucene.store.FSDirectory$FSIndexInput.&lt;init&gt;(FSDirectory.java:410)
      at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.&lt;init&gt;(NIOFSDirectory.java:123)
      at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80)
      at org.apache.lucene.store.RateLimitedFSDirectory.openInput(RateLimitedFSDirectory.java:99)
      at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:436)
      at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:318)
      at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:380)
      at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:812)
      at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:663)
      at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:376)
      at org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:947)
      at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:502)
      at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:186)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)
  </code></pre>
  
  <p>Here's the output from my node API:</p>
  
  <pre><code>    {
        "ok" : true,
        "cluster_name" : "Odine",
        "nodes" : {
          "J_vVn1gKQMmcDYADbcPlfw" : {
            "name" : "rap-es-1",
            "transport_address" : "inet[/141.161.88.174:9300]",
            "hostname" : "rap-es-1.uis.georgetown.edu",
            "version" : "0.90.3",
            "http_address" : "inet[/141.161.88.174:9200]",
            "os" : {
              "refresh_interval" : 1000,
              "available_processors" : 4,
              "cpu" : {
                "vendor" : "Intel",
                "model" : "Xeon",
                "mhz" : 1995,
                "total_cores" : 4,
                "total_sockets" : 1,
                "cores_per_socket" : 4,
                "cache_size" : "18kb",
                "cache_size_in_bytes" : 18432
              },
              "mem" : {
                "total" : "7.6gb",
                "total_in_bytes" : 8255045632
              },
              "swap" : {
                "total" : "7.9gb",
                "total_in_bytes" : 8589926400
              }
            },
            "process" : {
              "refresh_interval" : 1000,
              "id" : 31729,
              "max_file_descriptors" : 65535
            }
          },
          "hT_Z-JneS6uCem0M-7nTEQ" : {
            "name" : "rap-es",
            "transport_address" : "inet[/141.161.88.169:9300]",
            "hostname" : "rap-es.uis.georgetown.edu",
            "version" : "0.90.3",
            "http_address" : "inet[/141.161.88.169:9200]",
            "attributes" : {
              "data" : "false"
            },
            "os" : {
              "refresh_interval" : 1000,
              "available_processors" : 2,
              "cpu" : {
                "vendor" : "Intel",
                "model" : "Xeon",
                "mhz" : 1995,
                "total_cores" : 2,
                "total_sockets" : 2,
                "cores_per_socket" : 1,
                "cache_size" : "18kb",
                "cache_size_in_bytes" : 18432
              },
              "mem" : {
                "total" : "7.6gb",
                "total_in_bytes" : 8255303680
              },
              "swap" : {
                "total" : "7.9gb",
                "total_in_bytes" : 8589926400
              }
            },
            "process" : {
              "refresh_interval" : 1000,
              "id" : 10282,
              "max_file_descriptors" : 500000
            }
          },
          "-76vB5ISTUq4M7ESizP7og" : {
            "name" : "rap-es-4",
            "transport_address" : "inet[/141.161.88.177:9301]",
            "hostname" : "rap-es-4.uis.georgetown.edu",
            "version" : "0.90.3",
            "http_address" : "inet[/141.161.88.177:9201]",
            "os" : {
              "refresh_interval" : 1000,
              "available_processors" : 4,
              "cpu" : {
                "vendor" : "Intel",
                "model" : "Xeon",
                "mhz" : 1995,
                "total_cores" : 4,
                "total_sockets" : 1,
                "cores_per_socket" : 4,
                "cache_size" : "18kb",
                "cache_size_in_bytes" : 18432
              },
              "mem" : {
                "total" : "7.6gb",
                "total_in_bytes" : 8255045632
              },
              "swap" : {
                "total" : "7.9gb",
                "total_in_bytes" : 8589926400
              }
            },
            "process" : {
              "refresh_interval" : 1000,
              "id" : 2462,
              "max_file_descriptors" : 65535
            }
          },
          "xssBGLKITpicDcxdeEjr4A" : {
            "name" : "rap-es-6",
            "transport_address" : "inet[/141.161.88.179:9300]",
            "hostname" : "rap-es-6.uis.georgetown.edu",
            "version" : "0.90.3",
            "http_address" : "inet[/141.161.88.179:9200]",
            "os" : {
              "refresh_interval" : 1000,
              "available_processors" : 4,
              "cpu" : {
                "vendor" : "Intel",
                "model" : "Xeon",
                "mhz" : 1995,
                "total_cores" : 4,
                "total_sockets" : 1,
                "cores_per_socket" : 4,
                "cache_size" : "18kb",
                "cache_size_in_bytes" : 18432
              },
              "mem" : {
                "total" : "7.6gb",
                "total_in_bytes" : 8255045632
              },
              "swap" : {
                "total" : "7.9gb",
                "total_in_bytes" : 8589926400
              }
            },
            "process" : {
              "refresh_interval" : 1000,
              "id" : 2608,
              "max_file_descriptors" : 65535
            }
          },
          "wg_I5wBfRxqs6kdH-S_kww" : {
            "name" : "rap-es-6",
            "transport_address" : "inet[/141.161.88.179:9301]",
            "hostname" : "rap-es-6.uis.georgetown.edu",
            "version" : "0.90.3",
            "http_address" : "inet[/141.161.88.179:9201]",
            "os" : {
              "refresh_interval" : 1000,
              "available_processors" : 4,
              "cpu" : {
                "vendor" : "Intel",
                "model" : "Xeon",
                "mhz" : 1995,
                "total_cores" : 4,
                "total_sockets" : 1,
                "cores_per_socket" : 4,
                "cache_size" : "18kb",
                "cache_size_in_bytes" : 18432
              },
              "mem" : {
                "total" : "7.6gb",
                "total_in_bytes" : 8255045632
              },
              "swap" : {
                "total" : "7.9gb",
                "total_in_bytes" : 8589926400
              }
            },
            "process" : {
              "refresh_interval" : 1000,
              "id" : 2570,
              "max_file_descriptors" : 65535
            }
          },
          "lG0sqr95Q02Lt-iTCcg6qg" : {
            "name" : "rap-es-5",
            "transport_address" : "inet[/141.161.88.178:9300]",
            "hostname" : "rap-es-5.uis.georgetown.edu",
            "version" : "0.90.3",
            "http_address" : "inet[/141.161.88.178:9200]",
            "os" : {
              "refresh_interval" : 1000,
              "available_processors" : 4,
              "cpu" : {
                "vendor" : "Intel",
                "model" : "Xeon",
                "mhz" : 1995,
                "total_cores" : 4,
                "total_sockets" : 1,
                "cores_per_socket" : 4,
                "cache_size" : "18kb",
                "cache_size_in_bytes" : 18432
              },
              "mem" : {
                "total" : "7.6gb",
                "total_in_bytes" : 8255045632
              },
              "swap" : {
                "total" : "7.9gb",
                "total_in_bytes" : 8589926400
              }
            },
            "process" : {
              "refresh_interval" : 1000,
              "id" : 22580,
              "max_file_descriptors" : 65535
            }
          },
          "biDQwwn4R9GTkesZUlNW1g" : {
            "name" : "rap-es-2",
            "transport_address" : "inet[/141.161.88.175:9300]",
            "hostname" : "rap-es-2.uis.georgetown.edu",
            "version" : "0.90.3",
            "http_address" : "inet[/141.161.88.175:9200]",
            "os" : {
              "refresh_interval" : 1000,
              "available_processors" : 4,
              "cpu" : {
                "vendor" : "Intel",
                "model" : "Xeon",
                "mhz" : 1995,
                "total_cores" : 4,
                "total_sockets" : 1,
                "cores_per_socket" : 4,
                "cache_size" : "18kb",
                "cache_size_in_bytes" : 18432
              },
              "mem" : {
                "total" : "7.6gb",
                "total_in_bytes" : 8255045632
              },
              "swap" : {
                "total" : "7.9gb",
                "total_in_bytes" : 8589926400
              }
            },
            "process" : {
              "refresh_interval" : 1000,
              "id" : 5190,
              "max_file_descriptors" : 65535
            }
          },
          "A7E7I7R5SIib6RJPA3RNug" : {
            "name" : "rap-es-4",
            "transport_address" : "inet[/141.161.88.177:9300]",
            "hostname" : "rap-es-4.uis.georgetown.edu",
            "version" : "0.90.3",
            "http_address" : "inet[/141.161.88.177:9200]",
            "os" : {
              "refresh_interval" : 1000,
              "available_processors" : 4,
              "cpu" : {
                "vendor" : "Intel",
                "model" : "Xeon",
                "mhz" : 1995,
                "total_cores" : 4,
                "total_sockets" : 1,
                "cores_per_socket" : 4,
                "cache_size" : "18kb",
                "cache_size_in_bytes" : 18432
              },
              "mem" : {
                "total" : "7.6gb",
                "total_in_bytes" : 8255045632
              },
              "swap" : {
                "total" : "7.9gb",
                "total_in_bytes" : 8589926400
              }
            },
            "process" : {
              "refresh_interval" : 1000,
              "id" : 2500,
              "max_file_descriptors" : 65535
            }
          },
          "oQG-yfURTTWZiFC6eOwjAg" : {
            "name" : "rap-es-3",
            "transport_address" : "inet[/141.161.88.176:9300]",
            "hostname" : "rap-es-3.uis.georgetown.edu",
            "version" : "0.90.3",
            "http_address" : "inet[/141.161.88.176:9200]",
            "os" : {
              "refresh_interval" : 1000,
              "available_processors" : 4,
              "cpu" : {
                "vendor" : "Intel",
                "model" : "Xeon",
                "mhz" : 1995,
                "total_cores" : 4,
                "total_sockets" : 1,
                "cores_per_socket" : 4,
                "cache_size" : "18kb",
                "cache_size_in_bytes" : 18432
              },
              "mem" : {
                "total" : "7.6gb",
                "total_in_bytes" : 8255045632
              },
              "swap" : {
                "total" : "7.9gb",
                "total_in_bytes" : 8589926400
              }
            },
            "process" : {
              "refresh_interval" : 1000,
              "id" : 1291,
              "max_file_descriptors" : 65535
            }
          }
        }
      }
  </code></pre>
  
  <p>Any idea why I'm still getting the error after I've updated the file limit to 65535/node (500,000 on one for a test).</p>