title: mmap vs O_DIRECT for random reads (what are the buffers involved?)
body: <p>I am implementing a disk based hashtable supporting large amount of keys (26+ million). The value is deserialized. Reads are essentially random throughout the file, values are less than the page size, and I am optimising for SSDs. Safety/consistency are not such huge issues (performance matters).</p>
  
  <p>My current solution involves using a <code>mmap()</code> file with <code>MADV_RANDOM | MADV_DONTNEED</code> set to disable prefetching by the kernel and only load data as needed on-demand.</p>
  
  <p>I get the idea that the kernel reads from disk to memory buffer, and I deserialize from there.</p>
  
  <p>What about <code>O_DIRECT</code>? If I call <code>read()</code>, I'm still copying into a buffer (which I deserialize from) so can I gain any advantage?</p>
  
  <p>Where can I find more info on the buffers involved with a <code>mmap()</code> file and calling <code>read()</code> on a file opened with <code>O_DIRECT</code>?</p>
  
  <p>I am not interested in read ahead or caching. It has nothing to offer for my use case.</p>