title: Is it possible to map an existing buffer to a new file?
body: <p>The idea is relatively simple, but I see some complications for implementations, so I'm wondering if it's even possible right now.</p>
  
  <ul>
  <li>An example of what I'd like to do is to generate some data in a
  buffer, <em>then</em> map the contents of this buffer to a file.  Instead of
  having the memory space virtually populated with the contents of the
  file, I'd like the contents of the original buffer to be transferred
  to the system cache (which should be a zero-copy operation) and
  dirtied immediately (which would flush the data out to disk
  eventually).</li>
  </ul>
  
  <p>Of course the complication I mentioned is that the buffer should be deallocated and unmapped (since the data is now under the responsibility of the system cache), and I don't know how to do that either.</p>
  
  <p>The important aspects are that:</p>
  
  <ul>
  <li>The program can control when the file is <s>created</s> linked.</li>
  <li>The program isn't required to anticipate the size of the file nor does it have to remap it as the dataset grows.  Instead it can realloc the initial buffer (using an efficient memory allocator for this) until it is satisfied (it knows for sure that the dataset won't grow anymore) before finally mapping it to the file.</li>
  <li>The data remains accessible through the <em>same</em> virtual memory address even <em>after</em> being mapped to the file, still <em>without</em> a single intra-memory copy.</li>
  </ul>
  
  <p>One assumption is that:</p>
  
  <ul>
  <li>We can use an arbitrary memory allocator (or memory management scheme in general) that can manage dynamic buffers more efficiently than mmap/mremap can for the memory space it manages, because the latter must deal with the filesystem to grow/shrink the file, which would always be slower.</li>
  </ul>
  
  <p>So, (1) are these requirements too constrained?  (2) Is this assumption correct?</p>
  
  <p>PS:  I had to arbitrarily pick the tags for this question, but I'm also interested in hearing how BSDs and Windows would do this.  Of course if the POSIX API allows to do this already, that would be great.</p>
  
  <p><strong>Update:</strong>  I call a buffer a space of private memory (private to the process/task in any OS with normal VMM) allocated in primary memory. The high-level goal involves generating a dataset of an arbitrary size using another input (in my case the network), then once it's generated, make it accessible for long periods of time (to the network and to the process itself), saving it to disk in the process.</p>
  
  <ul>
  <li>If I keep the datasets in private memory and write them out normally, they'll just be swapped when the OS needs the space, which is a bit stupid since they're already on disk.</li>
  <li>If I map another region then I have to copy the contents of the buffer to that region (which resides in the system cache), which, again, is a tad stupid since I won't use that buffer after that.</li>
  </ul>
  
  <p>The alternative that I see is to write or use a full-blown userland cache reading and writing to the disk itself to ensure that (a) pages don't get uselessly swapped out and (b) the process doesn't hold too much memory for itself, which is never possible to do optimally anyway (better let the kernel do its job), and which is simply not a road I think is worth going down (too complex for less gains).</p>
  
  <p><strong>Update:</strong> Requirements 2 and 3 are non-issues considering Nominal Animal's answer.  Of course this implies that the assumption is incorrect, as he proved is almost the case (overhead is minimal).  I also relaxed requirement 1, <code>O_TMPFILE</code> is indeed perfect for this.</p>
  
  <p><strong>Update:</strong> <a href="https://lwn.net/Articles/591723/" rel="nofollow">A recent article on LWN</a> mentions, somewhere in the middle: <em>"That could possibly be done with a special write operation that would not actually cause I/O, or with a system call that would transfer a physical page into the page cache"</em>. That suggests that indeed, there is currently (April 2014) no way to do this at least with Linux (and likely other operating systems), much less with a standard API. The article is about PostgreSQL, but the issue in question is identical, except perhaps for the specific requirements to this question, which aren't defined in the article.</p>