title: Too many open files when opening and closing large numbers of files
body: <p>I am running some code which opens multiple files to read data, processes it and then closes the files through an experiment harness. The harness changes some parameters each time one lot of processing is complete and then runs the same code again with the updated parameters. </p>
  
  <p>When I run a small number of experiments there are no issues, and when the program exits all memory is freed. However, if I try to run a large number of experiments, the code always crashes after around 1000 calls to the processing code. I have checked that each file I open with <code>fopen</code> is closed using <code>fclose</code> in each individual part of the system, but the issue still occurs, with Valgrind outputting</p>
  
  <pre><code>==11892== Warning: invalid file descriptor 1030 in syscall open()
  ==11892== Warning: invalid file descriptor 1030 in syscall dup()
  Could not access specified file: Too many open files
  </code></pre>
  
  <p>after around 1000 iterations, and <code>perror</code> giving the "Too many open files" error. Is there something I have to do to reset the file descriptor count in order to open more files?</p>
  
  <p>edit: Looking at the <code>lsof -p</code> output during runtime, it seems that the type of files being opened is <code>DIR</code>. The files that I am accessing are inside different directories. I'm currently opening files with <code>fopen("path/to/directory/file.txt", "w")</code>. Is there something additional that I need to do to release the file descriptors that I've used?</p>
  
  <p>Here's an excerpt of the <code>lsof</code> output:</p>
  
  <pre><code>lt-launch 12101 michal   25r   DIR    8,1     4096 3537225 /directory/reference/20
  lt-launch 12101 michal   26r   DIR    8,1     4096 3537238 /directory/reference/21
  lt-launch 12101 michal   27r   DIR    8,1     4096 3537251 /directory/reference/22
  </code></pre>
  
  <p>edit2: The offending code is as follows:</p>
  
  <pre><code>FILE *fp;
  
  printf("Retrieving event data from file %s\n", filename);
  
  if ((fp = fopen(filename, "r")) == NULL){
  perror("Could not access specified file");
  exit(1);
  }
  
  char* line = malloc(MAX_LINE_LENGTH);
  char* lref = line;
  double_arr* event_times = init_double_arr(DEFAULT_ARR_SIZE);
  
  // Get data from file...
  
  if (i == 0){// we found no events in the given interval
  free_double_arr(event_times);
  free(lref);
  fclose(fp);
  return NULL;
  }
  event_times-&gt;data = realloc(event_times-&gt;data, i * sizeof(double));
  event_times-&gt;len = i;
  fclose(fp);
  free(lref);
  
  return event_times;
  </code></pre>
  
  <p>edit3: Although this point is where the program crashes, the cause is actually erroneous directory opening elsewhere in the code. See my answer below for what the issue was.</p>