title: "Too many open files in system" failure while listing a recursive directory structure
body: <p>I've implemented (in Java) a fairly straightforward Iterator to return the names of the files in a recursive directory structure, and after about 2300 files it failed "Too many open files in system" (the failure was actually in trying to load a class, but I assume the directory listing was the culprit).</p>
  
  <p>The data structure maintained by the iterator is a Stack holding the contents of the directories that are open at each level.</p>
  
  <p>The actual logic is fairly basic: </p>
  
  <pre><code>private static class DirectoryIterator implements Iterator&lt;String&gt; {
  
          private Stack&lt;File[]&gt; directories;
          private FilenameFilter filter;
          private Stack&lt;Integer&gt; positions = new Stack&lt;Integer&gt;();
          private boolean recurse;
          private String next = null;
  
          public DirectoryIterator(Stack&lt;File[]&gt; directories, boolean recurse, FilenameFilter filter) {
              this.directories = directories;
              this.recurse = recurse;
              this.filter = filter;
              positions.push(0);
              advance();
          }
  
          public boolean hasNext() {
              return next != null;
          }
  
          public String next() {
              String s = next;
              advance();
              return s;
          }
  
          public void remove() {
              throw new UnsupportedOperationException();
          }
  
          private void advance() {
              if (directories.isEmpty()) {
                  next = null;
              } else {
                  File[] files = directories.peek();
                  while (positions.peek() &gt;= files.length) {
                      directories.pop();
                      positions.pop();
                      if (directories.isEmpty()) {
                          next = null;
                          return;
                      }
                      files = directories.peek();
                  }
                  File nextFile = files[positions.peek()];
                  if (nextFile.isDirectory()) {
                      int p = positions.pop() + 1;
                      positions.push(p);
                      if (recurse) {
                          directories.push(nextFile.listFiles(filter));
                          positions.push(0);
                          advance();
                      } else {
                          advance();
                      }
                  } else {
                      next = nextFile.toURI().toString();
                      count++;
                      if (count % 100 == 0) {
                          System.err.println(count + "  " + next);
                      }
                      int p = positions.pop() + 1;
                      positions.push(p);
                  }
              }
          }
      }
  </code></pre>
  
  <p>I would like to understand how many "open files" this requires. Under what circumstances is this algorithm "opening" a file, and when does it get closed again?</p>
  
  <p>I've seen some neat code using Java 7 or Java 8, but I'm constrained to Java 6.</p>