title: Writing programs to cope with I/O errors causing lost writes on Linux
body: <p>TL;DR: <strong>If the Linux kernel loses a buffered I/O write</strong>, is there any way for the application to find out?</p>
  
  <p><strong>I know you have to <code>fsync()</code> the file (and its parent directory) for durability</strong>. The question is <em>if the kernel loses dirty buffers that are pending write</em> due to an I/O error, how can the application detect this and recover or abort?</p>
  
  <p>Think database applications, etc, where order of writes and write durability can be crucial.</p>
  
  <h2>Lost writes? How?</h2>
  
  <p>The Linux kernel's block layer can under some circumstances <em>lose</em> buffered I/O requests that have been submitted successfully by <code>write()</code>, <code>pwrite()</code> etc, with an error like:</p>
  
  <pre><code>Buffer I/O error on device dm-0, logical block 12345
  lost page write due to I/O error on dm-0
  </code></pre>
  
  <p>(See <a href="http://lxr.free-electrons.com/source/fs/buffer.c?v=2.6.26#L445" rel="noreferrer"><code>end_buffer_write_sync(...)</code> and <code>end_buffer_async_write(...)</code> in <code>fs/buffer.c</code></a>).</p>
  
  <p><a href="http://lxr.free-electrons.com/source/fs/buffer.c?v=4.9#L353" rel="noreferrer">On newer kernels the error will instead contain "lost async page write"</a>, like:</p>
  
  <pre><code>Buffer I/O error on dev dm-0, logical block 12345, lost async page write
  </code></pre>
  
  <p>Since the application's <code>write()</code> will have already returned without error, there seems to be no way to report an error back to the application.</p>
  
  <h2>Detecting them?</h2>
  
  <p>I'm not that familiar with the kernel sources, but I <em>think</em> that it sets <code>AS_EIO</code> on the buffer that failed to be written-out if it's doing an async write:</p>
  
  <pre><code>    set_bit(AS_EIO, &amp;page-&gt;mapping-&gt;flags);
      set_buffer_write_io_error(bh);
      clear_buffer_uptodate(bh);
      SetPageError(page);
  </code></pre>
  
  <p>but it's unclear to me if or how the application can find out about this when it later <code>fsync()</code>s the file to confirm it's on disk.</p>
  
  <p>It looks like <a href="http://lxr.free-electrons.com/source/mm/filemap.c?v=2.6.26#L267" rel="noreferrer"><code>wait_on_page_writeback_range(...)</code> in <code>mm/filemap.c</code></a> might by <a href="http://lxr.free-electrons.com/source/fs/sync.c?v=2.6.26#L250" rel="noreferrer"><code>do_sync_mapping_range(...)</code> in <code>fs/sync.c</code></a> which is turn called by <a href="http://lxr.free-electrons.com/source/fs/sync.c?v=2.6.26#L177" rel="noreferrer"><code>sys_sync_file_range(...)</code></a>. It returns <code>-EIO</code> if one or more buffers could not be written.</p>
  
  <p>If, as I'm guessing, this propagates to <code>fsync()</code>'s result, then if the app panics and bails out if it gets an I/O error from <code>fsync()</code> and knows how to re-do its work when restarted, that should be sufficient safeguard? </p>
  
  <p>There's presumably no way for the app to know <em>which</em> byte offsets in a file correspond to the lost pages so it can rewrite them if it knows how, but if the app repeats all its pending work since the last successful <code>fsync()</code> of the file, and that rewrites any dirty kernel buffers corresponding to lost writes against the file, that should clear any I/O error flags on the lost pages and allow the next <code>fsync()</code> to complete - right?</p>
  
  <p>Are there then any other, harmless, circumstances where <code>fsync()</code> may return <code>-EIO</code> where bailing out and redoing work would be too drastic?</p>
  
  <h2>Why?</h2>
  
  <p>Of course such errors should not happen. In this case the error arose from an unfortunate interaction between the <code>dm-multipath</code> driver's defaults and the sense code used by the SAN to report failure to allocate thin-provisioned storage. But this isn't the only circumstance where they <em>can</em> happen - I've also seen reports of it from thin provisioned LVM for example, as used by libvirt, Docker, and more. An critical application like a database should try to cope with such errors, rather than blindly carrying on as if all is well.</p>
  
  <p>If the <em>kernel</em> thinks it's OK to lose writes without dying with a kernel panic, applications have to find a way to cope.</p>
  
  <p>The practical impact is that I found a case where a multipath problem with a SAN caused lost writes that landed up causing database corruption because the DBMS didn't know its writes had failed. Not fun.</p>