title: stxxl sorting of very large file (ubuntu)
body: <p>I am trying to sort a large file with about billion records (each containing four integers). The size of the file would shoot up beyond 50GB.</p>
  
  <p>I am testing my code with 400 million records (about 6 GB file). My disk configuration looks like this: </p>
  
  <pre><code>disk=/var/tmp/stxxl,50G,syscall delete
  </code></pre>
  
  <p>My machine has 16 GB RAM and has 8 physical processors (Intel i7), <a href="http://stxxl.sourceforge.net/tags/master/index.html" rel="nofollow">stxxl</a> version 1.4.1. If I run the code with 200 million records, it takes about 5 minutes. But when I run the code with 400 million records, it seems to be running out of disk space. My questions are:</p>
  
  <p>1) Why is my code running out of disk space for sorting even a 6 GB file? Kindly review it (only few important lines attached).</p>
  
  <p>2) Is 5 minute a reasonable time for my PC to sort 200 million records? If its true, I wonder if stxxl can sort 5 billion records within daytime.</p>
  
  <p>3) Do you think stxxl is a nice choice for this sort of problem? I have access to a cluster as well which has mpi installed. </p>
  
  <p>CODE (inspired by examples/algo/sort_file.cpp and examples/algo/phonebills.cpp):</p>
  
  <pre><code>size_t memory_to_use = (1*1024) * 1024 * 1024ul;
  typedef stxxl::vector&lt;my_type, 1, stxxl::lru_pager&lt;8&gt;, block_size&gt; vector_type;
  
  std::copy(std::istream_iterator&lt;my_type&gt;(in),
    std::istream_iterator&lt;my_type&gt;(),
    std::back_inserter(v));
  
  stxxl::sort(v.begin(), v.end(), Cmp(), memory_to_use);
  </code></pre>
  
  <p>Each vector element or record is a tuple of four unsigned numbers:</p>
  
  <pre><code>struct my_type
  {
    typedef unsigned short key_type;
    typedef std::tuple&lt;key_type, key_type, key_type, key_type&gt; key4tuple;
    ...
  }
  </code></pre>