title: Improve throughput writing a lot of small files in C
body: <p>I want to improve the throughput of a software that writes, several usually small, files into a network attached volume.</p>
  
  <p>The volume is limited to 100 IOPS and 80 MB/s of bandwidth.</p>
  
  <p>At the moment I saturate the 100 IOPS but the bandwidth is very very far from the 80 MB/s reachable, ~4 MB/s but even less.</p>
  
  <p>I believe that the main issue is that we make a lot of small requests, those small requests saturated the IOPS but the bandwidth is pretty much left unexploited.</p>
  
  <p>The software is written in C and I control pretty much everything down to the actual <code>write</code> syscall.</p>
  
  <p>At the moment the architecture is multithreading, with several threads working as "spoolers" and making synchronous <code>write</code> call, each for a different file.</p>
  
  <p>So suppose we have file <code>a</code>, <code>b</code> and <code>c</code> and thread <code>t1</code>, <code>t2</code> and <code>t3</code>.</p>
  
  <p><code>t1</code> will open <code>a</code> and call in a loop something like <code>write(fd_a, buff_a, 1024)</code> and the same will do <code>t2</code> (<code>write(fd_b, buff_b, 1024)</code>) and <code>t3</code> (<code>write(fd_c, buff_c, 1024)</code>).</p>
  
  <p>Each file is a new file, so it get created at the first write.</p>
  
  <p>I believe that the problem is that the requests the OS is making (after the Linux IO scheduler merge) are pretty small, in the order of 10/20 blocks (5/10 kilobyte) each.</p>
  
  <p>The only way I see to fix the issue is to make bigger requests, but each file is small so I am not quite sure what is the best way forward.</p>
  
  <p>A possible idea could be to make a single <code>write</code> request instead of a loop of several request, so lookup how big is the file, allocate enough memory, populate the buffer and finally execute a single <code>write</code>.</p>
  
  <p>Another idea could be to switch so async io, but I don't have understand what the advantages would be in this case.</p>
  
  <p>Do you have any other suggestion?</p>